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Transitioning  to  a  New  Model? 

First  Consider  Your  Organizational  Culture 


The  decision  to  transition  my  organization  from  the  Capability  Maturity  Model®  for 
Software  (SW-CMM®)  to  CMM  Integration  (CMMI®)  was  not  an  easy  one.  We  had 
spent  the  previous  decade  immersed  in  change,  growing  from  an  ad-hoc  organization 
to  becoming  the  first  Department  of  Defense  software  engineering  house  to  achieve 
full  SW-CMM  Level  5  compliance.  It  has  been  a  rewarding  and  arduous  journey. 
However,  the  advent  of  CMMI  led  us  to  the  proverbial  crossroads  in  the  path  we  were 
traveling:  Do  we  plunge  in  and  incorporate  the  additional  CMMI  practices  into  our 
organizational  processes  and  in  so  doing  stay  at  the  leading  edge  of  process  maturity?  Or  do  we 
simply  utilize  the  robust  software  process  improvement  infrastructure  we  had  developed  to 
incrementally  get  better  and  better  over  time?  The  last  question  was  actually  a  compelling  argu¬ 
ment  against  transitioning  to  the  CMMI  model.  After  all,  the  ability  to  affect  continuous  process 
improvement  was  the  reason  we  adopted  the  Level  4  and  5  key  practices  of  the  SW-CMM  in  the 
first  place. 

In  the  end,  the  concern  over  the  planned  obsolescence  of  the  SW-CMM  model  and  the  addi¬ 
tional  benefits  of  CMMI  swung  the  debate  in  favor  of  the  transition.  I  knew  it  was  the  right 
thing  to  do,  but  nonetheless  had  apprehensions  about  leading  our  team  in  that  direction.  How 
would  this  decision  be  received  by  my  senior  staff  and  our  sizeable  work  force?  I  feared  it  could 
be  viewed  as  more  change  for  change’s  sake,  or  simply  as  an  attempt  to  keep  up  with  the  latest 
management  fad.  If  handled  incorrectly,  I  could  create  an  entire  organization  of  process 
improvement  cynics! 

I  knew  that  I  needed  to  communicate  our  goals  and  objectives  to  every  employee.  So  I  vis¬ 
ited  with  each  unit  and  explained  how  we  had  benefited  from  the  SW-CMM,  why  we  needed  to 
transition  to  CMMI,  how  I  expected  CMMI  would  benefit  us,  how  we  planned  to  make  the  tran¬ 
sition,  and  how  the  transition  would  affect  each  of  them  as  individuals.  I  can’t  say  that  everyone 
was  excited  about  making  the  change,  but  everyone  understood  why  we  were  doing  it;  they  sup¬ 
ported  it  and  made  it  happen.  More  importantly,  I  discovered  that  we  had  developed  something 
very  special  over  the  previous  10  years  of  incorporating  model-based  process  improvement:  We 
institutionalized  an  evolutionary  change  of  sorts  across  our  organization,  developing  the  mod¬ 
ern-day  survival  equivalent  of  the  chameleon’s  ability  to  adapt  to  changing  environments.  We 
had  created  an  organizational  culture  that  understands  and  accepts  change.  That  may  ultimate¬ 
ly  prove  to  be  one  of  the  greatest  values  to  us  that  have  resulted  from  our  SW-CMM,  and  now 
CMMI,  compliance. 

Dr.  Kenneth  Shere  observes  in  his  article,  Managing  Cultural  Changes  in  Your  Organisation ,  that 
process  improvement  causes  cultural  change,  change  is  hard,  and  that  one  must  have  a  com¬ 
pelling  reason  for  change.  I  encourage  you  to  take  advantage  of  his  recommendations  when 
embarking  on  your  next  organizational  change  adventure. 

Another  recent  change  being  accepted  by  my  organization  affects  CROSSTALK  as  I  move 
Tracy  Stauder  from  the  CROSSTALK  publisher  position  into  a  supervisory  position.  Tracy  has 
supported  CROSSTALK  since  1996  when  she  took  the  lead  in  its  production.  In  cooperation 
with  our  organization’s  desire  to  implement  consistent  CMM  processes,  Tracy  formalized  and 
documented  CROSSTALK’S  production  processes.  Those  who  have  been  subscribing  to 
CROSSTALK  since  before  she  joined  might  have  noticed  that  as  a  result  of  her  leadership, 
CROSSTALK’S  production  became  much  more  predictable  and  the  quality  also  improved. 

These  improvements  reflect  Tracy’s  outstanding  contributions  to  my  organization.  Her  lead¬ 
ership  talents  will  benefit  her  new  group,  and  this  opportunity  will  reward  her  excellent  work. 
One  basic  concept  of  process  maturity  is  that  by  having  established  processes  in  place,  an  orga¬ 
nization  will  not  depend  on  one  hero  to  keep  it  running.  Tracy  was  a  hero,  but  was  also  true  to 
our  process  improvement  initiatives,  which  will  ensure  CROSSTALK’S  continued  quality  in  this 
evolutionary  change. 


a  /A 

Randy  B.  Hill 

Ogden  Mir  Logistics  Center,  Co-Sponsor 
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Army  Simulation  Program  Balances  Agile  and 
Traditional  Methods  With  Success 


LTC  John  Surdu,  Ph.D.  DougJ.  Parsons 

U.S.  Army  Program  Executive  Office  —  Simulation  Training  and  Instrumentation 

The  One  Semi-Automated  Tones  (OneSAF)  Objective  System  is  the  next  generation  simulation  system  planned  to  provide 
the  U.S.  Army  with  an  entity-level  simulation  to  serve  three  modeling  and  simulation  domains.  Software  development  of  the 
OneSAF  application  has  been  conducted  in  a  highly  robust  systems  engineering  environment  based  on  commercial  and  gov¬ 
ernment  best  practices.  The  OneSAF  program  has  tailored  techniques  of  Extreme  Programming  (XP)  and  other  agile  meth¬ 
ods  into  a  development  environment  that  has  resulted  in  several  industry  awards ,  most  recently  the  National  Training  Systems 
Association  Cross  Function  Award  for  the  Integrated  Product  Team.  These  externally  certified  Capability  Maturity  Model m 
Integration  Level  5  processes  are  credited  with  successful  program  execution.  This  article  will  discuss  which  XP  and  other 
agile  techniques  were  used \  which  were  not ,  and  why. 


Principles  and  practices  associated  with 
agile  methods  are  not  new  concepts  in 
the  world  of  software  development. 
Extreme  Programming  (XP),  an  agile 
method  itself,  has  been  used  successfully 
in  a  variety  of  software  development  envi¬ 
ronments.  Overall  reviews  in  the  software 
development  community  have  been 
mixed.  In  his  history  of  the  Agile 
Manifesto  [1],  Jim  Highsmith  uses  terms 
like  organisational  anarchists  and  independent 
thinkers  to  describe  the  alliance  embracing 
a  unifying  set  of  values.  The  following  is 
stated  in  the  Agile  Manifesto: 

We  are  uncovering  better  ways  of 
developing  software  by  doing  it 
and  helping  others  do  it.  Through 
this  work  we  have  come  to  value: 

•  Individuals  and  interactions 
over  processes  and  tools. 

•  Working  software  over  com¬ 
prehensive  documentation. 

•  Customer  collaboration  over 
contract  negotiation. 

•  Responding  to  change  over  fol¬ 
lowing  a  plan. 

That  is,  while  there  is  value  in  the 
items  on  the  right,  we  value  the 
items  on  the  left  more.  [2] 

Pure  software  development  tradition¬ 
alists  consider  the  techniques  that  many 
agile  software  developers  use  to  follow 
these  values  as  an  excuse  to  hack  undocu¬ 
mented,  poorly  designed  code.  The  pure 
agile  developer  would  consider  the  plan- 
driven  traditionalists  responsible  for  sad¬ 
dling  software  developers  with  low-value 
processes  that  hinder  or  prevent  the  deliv¬ 
ery  of  software.  Fortunately,  we,  as  soft¬ 
ware  developers,  do  not  need  to  take  sides. 

®  Capability  Maturity  Model  is  registered  in  the  U.S.  Patent 
and  Trademark  Office  by  Carnegie  Mellon  University. 


There  exist  many  shades  of  gray  in  the 
spectrum  between  agile  and  traditional 
methods.  The  shade  you  select  that  best 
fits  your  program  depends  upon  many 
factors,  including  (but  not  limited  to)  team 
size,  criticality  of  defects,  ability  to  receive 
user  feedback/interaction,  customer 
expectations,  budget,  schedule,  and  stabil- 

“There  exist  many 
shades  of  gray  in  the 
spectrum  between  agile 
and  traditional  methods 
...  no  two  programs 
should  have  identical 
strategies  in  their 
software  development/9 

ity  of  requirements.  Since  these  factors  are 
different  from  program  to  program,  no 
two  programs  should  have  identical  strate¬ 
gies  in  their  software  development. 

Traditional  methods  are  geared  toward 
optimization,  predictability,  and  control. 
Agile  methods  focus  on  adaptation  to 
change,  flexibility,  and  innovation.  The 
new  art  of  software  development  is  find¬ 
ing  the  appropriate  balance  point  among 
the  available  practices.  Figure  1  is  a  chart 
developed  by  McCabe  and  Polen  [3]  based 
on  a  figure  from  Alistair  Cockburn  [4] 
indicating  the  relative  nature  between  agile 
and  conventional  projects  based  on  pro¬ 
ject  team  size  and  notional  cost  of  system 
failure.  Included  on  the  chart  is  the  place¬ 
ment  of  a  U.S.  Army  simulation  program, 
known  as  the  One  Semi-Automated 


Forces  (OneSAF)  Objective  System  (OOS). 

The  OOS  is  a  large  software-intensive 
system  (greater  than  two  million  lines  of 
source  code)  that  will  be  used  to  support 
research  and  development  and  train  future 
U.S.  military  leaders.  In  the  genesis  of  the 
OOS  program  development,  we  research¬ 
ed  and  considered  aspects  of  agile  meth¬ 
ods  and  XP  as  well  as  traditional  strategies. 
None  of  these  approaches  were  looked  on 
as  cookbook  methodologies,  but  rather  as 
a  smorgasbord.  Those  that  appeared  to  be 
a  good  fit  were  implemented;  others  were 
left  on  the  table. 

After  more  than  four  years  of  devel¬ 
opment,  the  OneSAF  program  remains 
on-track  to  satisfy  requirements  and  meet 
user  needs.  The  OneSAF  program  has 
been  awarded  the  National  Training  Sys¬ 
tems  Association  Cross  Function  Award 
for  the  Integrated  Product  Team;  it  was 
also  selected  by  CROSSTALK  as  one  of 
the  2003  U.S.  Government’s  Top  5  Quality 
Software  Projects. 

Interestingly  enough,  the  customer 
base  for  OOS  was  more  comfortable  with 
traditional  pedantic  software-development 
methods,  even  though  they  contributed  to 
an  environment  with  ill-defined  and  often- 
changing  requirements.  The  program  spent 
a  great  deal  of  time  educating  the  user  rep¬ 
resentatives  on  XP  and  other  agile  meth¬ 
ods.  Most  have  grown  to  embrace  these 
methods,  but  the  program  still  hears  some 
users  making  statements  like,  “Spiral  devel¬ 
opment  doesn’t  work;  it  has  created  a  con¬ 
figuration  management  nightmare,”  or 
“XP  has  reduced  time  for  testing.”  Despite 
statements  like  these,  the  processes  estab¬ 
lished  for  OOS  development  -  a  blend  of 
agile,  extreme,  and  traditional  techniques  - 
have  been  instrumental  in  the  program 
reaching  programmatic  and  technical  goals. 

This  article  is  not  intended  to  discuss 
the  technical  capabilities  of  the  OOS; 
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Figure  1 :  Spectrum  for  Project  Type  Considering  Defect  Cost  and  Team  Siye 


however,  it  is  important  that  the  reader 
understands  some  of  the  program’s  histo¬ 
ry  to  note  similarities  and  distinctions  with 
their  own  programs,  present  or  past. 

Mission  Need  and  the  Users 

The  OOS  is  the  Army’s  next  generation 
simulation  system  that  can  represent  a  full 
range  of  military  operations,  systems,  and 
control  processes.  It  is  an  entity-level  sim¬ 
ulation,  meaning  that  it  can  simulate  the 
activities  of  individual  combatants  or 
vehicles  (as  opposed  to  aggregate-level 
simulations,  which  represent  combatants 
and  vehicles  as  groupings).  It  will  also  pro¬ 
vide  the  appropriate  representations  of 
the  physical  environment  (e.g.,  terrain  fea¬ 
tures,  weather,  and  illumination)  and  its 
effect  on  simulated  activities  and  behav¬ 
iors. 

The  OOS  is  unique  among  Army  sim¬ 
ulations  in  that  it  is  designed  for  use  by 
three  distinct  Army  Modeling  and  Simu¬ 
lation  (M&S)  domains.  Specifically,  the 
Advanced  Concepts  and  Requirements 
domain  uses  M&S  for  experimentation 
and  analysis  on  Army  doctrine  and  force- 
related  concepts.  The  Research,  Develop¬ 
ment,  and  Acquisition  domain  uses  M&S 
for  acquisition  analyses  focused  on  equip¬ 
ping  and  supporting  currently  fielded  and 
future  forces.  Finally,  the  Training,  Exer¬ 
cises,  and  Military  Operations  domain 
employs  M&S  to  train  forces.  It  does  so 
using  live  simulation  (actual  equipment  on 
training  ranges),  virtual  simulation 
(immersing  the  trainee  into  a  synthetic 
environment),  and  constructive  simulation 
(war  games  using  computer-generated 
forces). 

There  exist  two  factors  relative  to  our 
users  that  inherently  pull  the  OOS  devel¬ 
opment  toward  the  center  of  the  agile- 
conventional  project  spectrum.  First, 
OOS  will  be  used  extensively  for  analysis 
and  experimentation.  Analysts  and  re¬ 
search  scientists  rely  on  a  robust  set  of 
documentation  to  support  verification  and 
validation  (V&V).  For  these  users,  it  is 
critical  to  understand  how  OOS  models 
work.  Secondly,  the  OneSAF  business 
model  includes  the  distribution  of  source 
code  with  release  of  the  software  baseline 
[5].  The  intention  is  to  create  an  environ¬ 
ment  that  will  optimize  the  ability  for 
extended  capabilities  created  by  the  M&S 
community  to  be  reintegrated  into  the 
baseline.  Agile  projects  tend  to  be  light  on 
documentation  and  process.  OOS  will 
include  a  robust  set  of  documentation  and 
tools  to  support  V&V.  In  addition, 
process  description  and  documentation  to 
aid  and  guide  external  developers  of  the 
baseline  will  also  be  provided. 


A  New  Approach 

While  the  use  of  commercial  best  prac¬ 
tices  seems  intuitive,  five  years  ago  this 
was  quite  unique  in  Department  of 
Defense  (DoD)  software  development. 
Some  of  the  changes  were  simple.  The 
approach  toward  programmatic  documen¬ 
tation  was  a  minimalist  one.  Some 
approaches  were  considered  heretical:  uti¬ 
lizing  the  OneSAF  program  manager 
(PM)  as  the  manager  of  the  OOS  task 
orders  (similar  to  a  prime  contractor ),  and 
establishing  an  integrated  development 
environment  (IDE),  which  made  the 
adoption  of  many  agile  methods  and  XP 
practices  possible. 

PM  Is  the  Prime 

A  typical  approach  in  DoD  software 
development  is  for  the  government  to 
select  a  single  contractor  who  specifies  its 
own  set  of  subcontractors  under  a  large, 
monolithic  contract.  In  contrast,  PM 
OneSAF  was  allowed  to  complete  a  vari¬ 
ety  of  task  orders  under  an  Indefinite 
Delivery,  Indefinite  Quantity  contract  and 
manage  it  as  the  lead  systems  integrator. 
The  advantage  is  that  the  government  can 
pick  a  best-of  breed  contractor  rather  than 
settle  for  a  sub-contractor  who  may  or 
may  not  be  the  best  choice.  During  the 
past  four  years,  26  different  contractor 
companies  have  been  contracted  to  work 
on  different  parts  of  the  OneSAF  soft¬ 
ware.  The  contracts  range  in  scope  and 
include  short-term  studies,  architecture 
review,  knowledge  acquisition,  architec¬ 
ture  and  integration,  model  development, 
and  integration  and  test. 

An  initial  concern  was  that  the  con¬ 
tractors  would  disagree  on  the  means  to 
resolve  issues  and  the  process  would  grind 
to  a  halt.  Associate  Contractor  Agree¬ 
ments  (ACAs)  were  signed  by  each  task 
order  organization;  however,  an  ACA  is 
only  a  piece  of  paper  that  asks  a  contrac¬ 
tor  to  play  nice.  To  help  socialise  the  ACAs, 
the  contracts  were  awarded  over  the  space 


of  18  months.  The  Architecture  and 
Integration  (A&I)  contract  was  awarded 
first,  and  the  processes,  tools,  and  proce¬ 
dures  were  established.  When  new  con¬ 
tractors  came  on  board,  they  were  inte¬ 
grated  into  existing  processes  so  that  we 
avoided  food  fights  about  whose  processes 
were  better  than  others.  This  does  not 
imply  that  existing  processes  were  never 
modified;  we  remained  committed 
throughout  execution  to  continuous 
process  improvement  and  aggressively 
sought  new  ideas. 

Three  factors  contributed  to  the  ability 
for  PM  OneSAF  to  successfully  act  as  the 
prime.  First,  the  OneSAF  government 
team  is  involved  in  the  day-to-day  develop¬ 
ment  process.  This  allows  informed  and 
timely  decisions  to  be  made  on  behalf  of 
the  PM.  Second,  PM  OneSAF  empowered 
the  A&I  contractor  with  a  great  deal  of 
flexibility  to  establish  development,  inte¬ 
gration,  and  test  processes  from  industry 
best  practices:  There  were  no  government- 
mandated  processes  heavy  with  valueless 
documentation.  Third,  the  PM  sought 
technically  qualified  folks  across  the 
breadth  of  the  program.  Not  only  are 
OneSAF  engineers  truly  technical  with 
advanced  degrees  in  software-related  disci¬ 
plines  and  years  of  real  engineering  experi¬ 
ence,  but  OneSAF  managers  and  project 
directors  were  recruited  from  engineering. 
As  there  are  numerous  disincentives  for 
technical  people  within  the  Army,  it  was 
quite  challenging  to  find  even  the  small 
number  of  qualified  engineers  we  needed. 

It  is  unclear  whether  these  unique  busi¬ 
ness  processes  alone  were  the  major  con¬ 
tributor  to  successful  execution  or  whether 
the  employment  of  agile  methods  was  the 
key.  It  is  clear,  however,  that  without  these 
new  ways  of  doing  business,  it  is  unlikely 
that  agile  methods  would  have  been 
employed  or  embraced. 

IDE 

To  support  an  effective  and  efficient  soft- 
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ware  development  process,  PM  OneSAF 
established  an  environment  that  brought 
together  domain/user  representatives, 
government  engineers,  PMs,  and  contract 
or  software  developers.  All  of  these  teams 
are  collocated  in  a  single  facility.  A  task 
order  for  facility  operation  and  sustain¬ 
ment  was  intentionally  awarded  to  a  con¬ 
tractor  outside  any  of  those  developing 
software  to  send  the  message  that  the  IDE 
was  neutral  turf.  Over  the  past  four  years, 
there  have  been  as  many  as  a  dozen  indi¬ 
vidual  task  orders  under  execution  at  any 
one  time.  From  these  task  orders,  there 
have  been  more  than  100  software  engi¬ 
neers  working  in  concert  to  develop  a 
baseline. 

Applying  Agile  Methods  and 

XP  to  OneSAF 

Communication  and  Collocation 

The  Agile  Manifesto  states  that  in  agile 
software  development,  the  following 
should  occur  [6]: 

•  Business  people  and  developers  must 
work  together  daily  throughout  the 
project. 

•  The  most  efficient  and  effective 
method  of  conveying  information  to 
and  within  a  development  team  is  face- 
to-face  conversation. 

Paulk  states  that  agile  methods  gener¬ 
ally  apply  to  smaller  teams  working  in  the 
face  of  vague  and/or  dynamic  require¬ 
ments  [7].  He  also  states  that  agile  teams 
are  expected  to  be  collocated  with  typical¬ 
ly  fewer  than  10  members.  XP  rules  also 
indicate  that  the  customer  should  always 
be  available  and  that  a  stand-up  meeting 
starts  each  day  to  communicate  problems, 
solutions,  and  promote  team  focus. 

OOS  development  has  practiced  collo¬ 
cation  from  its  inception.  As  mentioned 
earlier,  the  government  program  office  is 
the  prime  for  OOS,  and  there  are  numer¬ 
ous  contractors  working  the  program.  All 
contractors  from  the  10  to  12  companies 
working  on  OOS  are  collocated  in  the 
IDE  facility.  In  addition,  the  combat 
developer  (requirements  steward  and  cus¬ 
tomer),  customer  representatives  (from 
the  three  Army  M&S  domains),  and  gov¬ 
ernment  engineers  and  managers  are  col¬ 
located  in  the  same  facility.  It  often  takes  a 
newcomer  to  the  program  months  to  find 
out  what  company  everyone  works  for 
because  on  Team  OneSAF  we  concentrate 


on  functional  decomposition  of  the  prob¬ 
lem  more  than  on  which  company  is 
working  a  portion  of  the  problem. 

Through  the  use  of  ACAs,  communi¬ 
cation  between  the  various  organizations 
is  smooth  and  seamless.  The  face-to-face 
coordination  referred  to  in  the  Agile 
Manifesto  occurs  habitually  in  the  IDE.  A 
member  of  the  combat  development 
team  said  recently,  “Much  of  the  work  on 
OOS  occurs  in  the  hallways.”  Informal 
meetings  with  the  right  two  or  three  peo¬ 
ple  in  the  hallway  often  work  through 
some  technical  or  inter-team  coordination 
issue  in  a  few  minutes  rather  than  sched¬ 
uling  a  meeting  with  dozens  of  folks. 
OOS  developers  are  encouraged  to  get  up 
and  walk  down  the  hallway  if  they  have  an 
issue.  From  the  earliest  days  of  OOS 
development,  the  OneSAF  PM  Office 
prohibited  weekly  scheduled  meetings  on 
Wednesday  through  Friday.  A  lesson 
learned  from  other  program  development 
efforts  was  that  engineers  tended  to  save 
communication  with  other  people  on  the 
team  until  the  scheduled  meeting. 
Intentionally,  this  forced  the  engineers  to 
meet  more  frequently  in  desk-side  or  hall¬ 
way  meetings. 

Spiral  Development :  Builds ,  Blocks, 
and  Early  Delivery 

In  discussing  requirements,  McCabe  and 
Polen  state: 

. . .  the  customer  may  hardly  grasp 
the  problem,  much  less  the  best 
system  to  address  it.  Therefore, 
requirements  are  likely  to  be  vague 
or  speculative  when  they  should  be 
specific.  [3] 

Recognizing  that  the  customers’  require¬ 
ments  and  desires  would  evolve  as  they 
saw  working  prototypes  (or  alpha  versions 
of  the  software),  OOS  adapted  a  spiral 
development  methodology.  Paulk  noted: 

. . .  with  their  emphasis  on  address¬ 
ing  requirements  volatility,  agile 
methodologies  could  be  a  powerful 
synthesis  of  practices  that  DoD 
contractors  could  leverage  to  make 
planning  more  responsive  to 
change.  [7] 

The  Agile  Manifesto  supports  the 


notion  of  spiral  development  with  the  fol¬ 
lowing  tenets  [6] : 

•  The  highest  priority  is  to  satisfy  the 
customer  though  early  and  continuous 
delivery  of  valuable  software. 

•  Welcome  changing  requirements,  even 
late  in  development. 

•  Deliver  working  software  frequently, 
from  a  couple  of  weeks  to  a  couple  of 
months,  with  a  preference  to  the 
shorter  timescale. 

•  Working  software  is  the  primary  mea¬ 
sure  of  progress. 

XP  rules  and  practices  concerning  the 
software  development  process  include  the 
following  [8]: 

•  Make  frequent  small  releases. 

•  The  project  is  divided  into  iterations. 

•  Integrate  often. 

•  Iteration  planning  starts  each  iteration. 

•  Never  add  functionality  early. 

•  Code  must  be  written  to  agreed-upon 
standards. 

The  development  methodology  adopt¬ 
ed  for  OOS  was  one  of  frequent  itera¬ 
tions,  or  spirals.  It  involved  breaking  the 
overall  program  into  a  series  of  eight-  to 
10-week  builds.  Several  of  these  builds 
were  then  designated  as  user  assessment 
baselines  that  were  made  available  to  users 
for  assessment  and  azimuth  correction. 
User  involvement  is  discussed  below. 

The  early  and  frequent  delivery  of 
builds  was  the  OOS’  way  of  implementing 
these  tenets  of  the  manifesto.  Being  able 
to  see  working  code  —  even  if  that  code 
initially  was  focused  on  architecture  and 
tools  rather  than  interesting  military  capa¬ 
bility  -  gave  the  users  confidence  in  the 
process  and  the  development  team.  As 
McCabe  and  Polen  state,  “Until  a  usable 
system  is  delivered,  the  customer  has 
nothing  to  show  for  its  investment”  [3]. 
Paulk  asserted  the  following: 

Agile  methodologies,  with  their 
rapid  iterations,  require  continual 
planning.  Customer  collaboration 
and  responsiveness  to  change  are 
tightly  linked,  if  perhaps  inconsis¬ 
tent  with  typical  government-con¬ 
tractor  relationships.  [7] 

The  overall,  strategic  program  requirements 
and  overall  architecture  changed  little  over 
the  four  years  of  software  development; 
however,  the  specific,  tactical  requirements 
and  many  of  the  smaller  design  decisions 
were  made  as  late  as  possible. 

The  overall  goals  of  a  particular  block 
were  locked  down  during  the  block  plan¬ 
ning  a  few  weeks  before  the  beginning  of 
the  block,  not  at  the  beginning  of  the  pro¬ 
gram  in  a  big  bang.  Additionally,  the  fine- 


Figure  2:  Three-Build  Process  for 
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grained  requirements  and  design  of  a  build 
were  finalized  only  one  build  before  execu¬ 
tion.  While  executing  Build  X,  the  program 
is  planning  Build  X+l  and  testing  Build  X- 
1.  A  change  in  requirements,  therefore, 
could  be  reacted  to  within  16  weeks,  not  a 
year.  Figure  2  illustrates  this  process. 

A  major  advantage  of  a  spiral  method¬ 
ology  was  the  ability  of  the  program  to 
adapt  to  requirements  changes.  McCabe 
and  Polen  asserted,  “When  needs  are 
changing,  the  value  of  the  original  system 
as  specified,  however  optimum  it  was  at 
the  time  of  its  conception,  depreciates 
daily”  [3].  During  program  development, 
the  Army  made  a  radical  change  in  organi¬ 
zation  to  a  focus  on  Brigade  Combat 
Teams.  The  OOS  program  was  able  to 
rapidly  shift  focus  between  spirals  so  that 
it  would  be  delivered  with  the  new  force 
structure  rather  than  the  old.  Additionally, 
the  threat  faced  by  U.S.  forces  changed. 
OOS  was  able  to  curtail  representation  of 
older,  Soviet-style  opposing  forces  and 
implement  a  more  contemporary,  uncon¬ 
ventional  enemy. 

Customer  Involvement  in 
Development 

Turner  and  Boehm  state: 

One  of  the  major  differences 
between  agile  and  plan-driven 
methods  is  that  agile  methods 
strongly  emphasize  having  dedicat¬ 
ed,  collocated  customer  represen¬ 
tatives,  while  plan-driven  methods 
count  on  a  good  deal  of  up-front, 
customer-developer  work  on  con¬ 
tractual  plans  and  specifications.  [9] 

Because  the  customer  representatives  are 
collocated  in  the  IDE,  they  participate  in 
all  our  meetings,  and  are  available  to 
answer  questions  or  reach  back  into  their 
customer  base  for  feedback  and  input. 

Several  of  the  OOS  builds  were  desig¬ 
nated  User  Assessment  Baselines  (UAB). 
These  UABs  were  available  in  the  IDE  for 
users  to  evaluate.  In  addition,  a  number  of 
assessment  events  were  held  in  the  IDE 
during  which  users  from  around  the  Army 
were  invited  to  participate.  Finally,  the  PM 
office  took  the  software  to  user  sites  for 
more  formal  assessments.  It  may  seem 
obvious  to  some,  but  users  use  the  soft¬ 
ware  differently  than  engineers  and  devel¬ 
opers.  While  the  users  often  tried  to  treat 
these  developmental  assessments  as  oper¬ 
ational  tests,  resulting  in  often  vitriolic 
feedback,  the  events  were  excellent  oppor- 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark 
Office  by  Carnegie  Mellon  University. 


tunities  to  allow  users  to  identify  bugs  in 
the  software  in  an  operational-like  envi¬ 
ronment. 

Documentation  Versus  Working  Code 

OOS  software  builds  can  vary  between 
eight  and  1 0  weeks  in  duration,  depending 
upon  the  difficulty  of  the  tasks  in  that 
build,  and  how  they  fit  in  the  timeframe  of 
the  block.  Since  Build  4,  OOS  has  had 
working  software  that  could  be  demon¬ 
strated  and  made  available  for  user  feed¬ 
back.  As  McCabe  and  Polen  state,  “Your 
only  real  knowledge  comes  from  a  work¬ 
ing  system”  [3]. 

Having  working  code  since  Build  4 
reflects  the  agile  method’s  bias  toward 
working  code  rather  than  documentation¬ 
centric  development.  However,  extensive 
design  documentation,  knowledge  acquisi¬ 
tion  documentation,  and  technical  notes 
do  exist  and  are  reposed  in  <www.One 
SAF.net>,  the  program’s  collaborative 

“Be/ng  able  to  see 
working  code  -  even  if 
that  code  initially  was 
focused  on  architecture 
and  tools  rather  than 
interesting  military 
capability  -  gave  the 
users  confidence  in  the 
process  and  the 
development  team.” 

information  warehouse.  In  addition,  user 
documentation  is  maintained  in  a  form 
that  is  simultaneously  compiled  into  a 
users’  manual  and  online  help. 

Paulk  states  that  when  employing  agile 
methods,  a  project  must: 

Decide  where  to  place  the  balance 
point  in  documentation  and  plan¬ 
ning  to  alleviate  the  concerns  of 
the  stakeholders  (and  regulatory 
requirements)  while  achieving  the 
flexibility  and  benefits  promised  in 
the  agile  philosophy.  [7] 

This  is  not  to  imply  that  the  OOS  soft¬ 
ware  is  not  adequately  documented.  The 
A&I  contractor  has  been  externally  certi¬ 
fied  at  Capability  Maturity  Model 
Integration  (CMMI®)  Level  5,  so  the 


development  processes  are  well  docu¬ 
mented.  Not  only  are  the  processes  docu¬ 
mented,  but  the  results  of  the  processes 
(the  development  products/artifacts)  are 
captured.  Metrics  are  captured  on  the  exe¬ 
cution  of  processes  as  well,  and  the  A&I 
contractor  conducts  periodic  defect  preven¬ 
tion  sessions  to  examine  and  correct  the 
root  cause  of  common  issues. 

The  documentation  of  these  processes 
and  products  is  captured  in  a  Web-enabled 
manual  known  as  the  Electronic  Process 
Guide.  The  artifacts  that  document  devel¬ 
opers’  adherence  to  these  processes  are 
found  in  several  Web-enabled  repositories 
such  as  the  online  Software  Development 
Folders,  Web-based  tracking  tools  for 
action  items,  trouble  reports,  defects,  peer 
reviews,  and  risks.  The  very  nature  of  hav¬ 
ing  a  Web-enabled  tool  set  reduced  the 
burden  on  developers  for  complying  with 
these  processes  and  enabled  communica¬ 
tion  across  teams. 

Extreme  Testing 

With  respect  to  testing,  agile  methods 
indicate  the  projects  should  include  the 
following  [8]: 

•  Code  the  unit  test  first. 

•  All  code  must  have  unit  tests  that  must 
pass  before  being  released. 

•  When  a  bug  is  found,  tests  are  created. 

•  Acceptance  tests  are  run  often  and  the 
score  is  published. 

Whenever  an  OOS  developer  commits  his 
or  her  changes  to  configuration  manage¬ 
ment  (i.e.,  Concurrent  Versions  System),  a 
tool  -  called  BuildBoy  —  builds  the  soft¬ 
ware  for  Linux,  Windows,  and  Solaris  and 
runs  nearly  3,000  automated  tests  on  each 
operating  system.  If  the  build  fails,  the 
developer  and  his  or  her  supervisor  are 
sent  an  e-mail.  The  developer  can  then  ref¬ 
erence  the  BuildBoy  Web  page  to  deter¬ 
mine  the  nature  of  the  failure  and  take  cor¬ 
rective  action.  In  this  way,  the  software  is 
built  and  subjected  to  unit  and  integration 
tests  on  average  of  eight  times  a  day. 
McCabe  and  Polen  state: 

Writing  automated  unit  tests  first  is 
a  clever  way  of  inducing  develop¬ 
ers  to  not  only  unit  test  their  code 
efficiently,  but  also  to  write  their 
code  efficiently  without  superflu¬ 
ous  logic.  [3] 

While  we  agree  in  concept,  this  is  an  area 
in  which  we  need  improvement.  All  code 
is  delivered  at  the  end  of  a  build  with 
appropriate  tests,  and  some  of  the  teams 
do  build  the  tests  first,  but  we  are  not  con¬ 
sistently  using  the  methodology  of  build¬ 
ing  the  test  first. 
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With  respect  to  the  tenet  of  building  a 
test  when  a  bug  is  found,  we  have  adapted 
this  process  to  large,  complex  software. 
While  we  do  not  build  a  test  for  every  bug 
that  is  fixed,  when  we  find  the  same  pat¬ 
tern  of  bug  appears  many  times,  we  build 
an  automated  test  to  trap  it. 

Conclusion 

The  OOS  software  is  being  developed 
through  a  combination  of  agile,  extreme, 
and  traditional  techniques.  We  have  not 
blindly  adopted  all  of  the  techniques  and 
tenets  of  these  agile  approaches;  however, 
we  have  used  a  great  many  of  them.  The 
program  still  has  room  for  improvement  in 
some  areas,  and  our  CMMI  Level  5  contin¬ 
uous  process  improvement  is  ever  vigilant 
for  opportunities  to  do  so.  In  particular,  we 
are  in  need  of  more  agile  testing  and  an 
increased  number  of  automated  tests. 

The  unique  blend  of  these  techniques 
has  been  instrumental  in  the  award-win¬ 
ning  success  of  the  OOS  development 
effort.  Some  readers  may  wonder  which  of 
these  techniques  were  most  beneficial  and 
if  they  could  be  applied  to  other  programs 
with  equal  success.  The  answer  to  both  of 
these  questions  is  based  upon  our  original 
premise:  the  best  method  depends  upon 
the  nature  of  the  program  itself.  Program 
characteristics  such  as  team  size  and 
impact  of  system  failure  are  evident;  how¬ 
ever,  each  PM  needs  to  consider  other 
issues  that  define  the  technical,  program¬ 
matic,  and  political  landscape:  Will  upper 
management  support  less  conventional 
techniques?  Does  the  government  team 
have  the  appropriate  skill  set  to  work 
closely  with  the  software  development 
team,  not  just  monitor  contractor  activity 
and  check  end  results?  Are  the  users  open 
to  agile  methods  and  willing  to  actively 
participate  throughout  the  process? 

The  best  methods  are  as  unique  as  the 
programs  themselves.  Since  our  program 
initiation,  we  have  engaged  with  other 
organizations  that  have  tried  to  emulate 
the  form  —  without  the  substance  —  of 
these  agile  methods  and  innovative  busi¬ 
ness  processes;  their  successes  have  been 
limited.  The  authors  considered  what 
methods  and  techniques  worked  best  for 
the  OneSAF  program.  After  pondering 
this  question,  the  answer  illustrates  the 
shades  of  gray  between  traditional  and 
agile  methods.  Strictly  following  the 
CMMI  Level  5  processes  (traditional)  and 
the  individual  interactions  (agile)  among 
users,  developers,  and  government  repre¬ 
sentatives  were  essential  contributors. 
Four  years  ago  we  did  not  know  what 
would  work  best,  or  what  would  work  at 
all  for  that  matter.  However,  we  were  will¬ 


ing  to  try  something  innovative,  willing  to 

make  changes  along  the  way,  and  bold 

enough  to  see  it  through. ♦ 
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Did  this  article  pique  your  interest? 

You  can  hear  more  at  the  Eighteenth 
Annual  Systems  and  Software  Technolo¬ 
gy  Conference  May  1-4,  2006,  in  Salt 
Lake  City,  UT.  Doug  Parsons  will  pre¬ 
sent  in  Track  7,  Room  251A-C  on 
Thursday,  May  4,  at  8  a.m. 
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Managing  Cultural  Changes  in  Your  Organization 

Dr.  Kenneth  D.  Shere 
The  Aerospace  Corporation 

Maturity  models  are  great;  they  provide  a  mostly  technical  road  map  for  what  we  need  to  do  to  improve  processes.  Tean  Six 
Sigma  is  also  great  —  it  provides  a  methodology  for  how  to  improve  processes.  The  problem  is  that  process  improvement  caus¬ 
es  cultural  changes.  This  article  provides  guidelines  for  managing  cultural  changes  in  your  organisation. 


In  earlier  articles  [1,  2],  discussions  on 
how  Lean  Six  Sigma  (LSS)  affects  the 
government,  and  how  LSS  compares  to 
the  Capability  Maturity  Model®  (CMM®) 
have  been  presented.  Key  aspects  of  why 
LSS  has  been  successful  include  the  fol¬ 
lowing: 

1.  It  has  an  external  focus  based  on  lis¬ 
tening  to  the  voice  of  the  customer. 

2.  The  cost  of  poor  quality  is  explicitly 
considered. 

3.  LSS  is  essentially  good  systems  engi¬ 
neering  in  which  the  methodology  is 
institutionalized. 

4.  Training  is  an  inherent  part  of  the 
methodology. 

Many  articles  on  capability  maturity  mod¬ 
els  have  been  published  in  CROSSTALK. 
The  internal  focus  and  structure  of  these 
are  complementary  to  LSS.  Both 
approaches  have  been  used  successfully, 
and  when  combined  provide  a  powerful 
basis  for  producing  effective  products 
with  low  time  and  budget  variances. 

This  claim  has  been  recognized.  For 
specific  examples,  the  reader  can  search 
the  Software  Engineering  Institute  Web 
site  at  <www.sei.cmu.edu>.  One  of  the 
companies  publishing  results  on  that  site 
is  Raytheon.  Specifically,  it  is  their  North 
Texas  Software  division,  located  in  Plano 
and  McKinney. 

While  these  methods  are  useful  and 
there  exists  substantial  data  from  both 
government  oriented  programs  and  pure¬ 
ly  commercial  programs  to  prove  it,  they 
inherently  involve  changing  the  way  orga¬ 
nizations  conduct  business. 

The  change  process  is  very  taxing. 
People  at  all  levels  of  an  organization  tend 
to  resist  change.  This  article  explores  why 
change  is  necessary,  and  provides  some 
information  on  what  it  takes  to  enact 
change.  This  article  is  intended  for  leaders 
of  organizations  who  are  grappling  with 
change.  The  topics  discussed  are  the  fol¬ 
lowing:  change  is  hard,  facilitating  change, 
strategic  thinking,  and  first-year  strategy. 
There  are  many  books  and  journal  articles 
written  on  these  topics.  A  high-level  dis¬ 
cussion  is  provided  here  to  provoke 
thought  and  additional  reading. 


Change  Is  Hard 

Change  affects  people’s  lives.  Reaction  to 
change  varies  from  the  view  expressed  by 
the  comic  strip  character,  Pogo,  “The  cer¬ 
tainty  of  misery  is  better  than  the  misery 
of  uncertainty,”  to  the  view  of  the  late 
Jerry  Garcia  of  the  rock  band  The 
Grateful  Dead,  “Somebody  has  to  do 
something,  and  it’s  incredibly  pathetic  that 
it  has  to  be  us”  [3].  To  successfully  imple¬ 
ment  change,  it  is  necessary  to  understand 


“To  successfully 
implement  change,  it  is 
necessary  to  understand 
the  urgency  for  change, 
and  to  adopt  a  positive 
attitude  about  change. 
Understanding  how 
change  affects  the 
people  in  an 
organization  is  a  key 
component  of  breaking 
down  resistance.9* 


the  urgency  for  change  and  to  adopt  a 
positive  attitude  about  change.  Under¬ 
standing  how  change  affects  people  in  an 
organization  is  a  key  component  of  break¬ 
ing  down  resistance. 

Some  of  the  causes  for  the  urgency  of 
change  could  be  the  following: 

•  Your  customers  are  dissatisfied  (and 
may  be  considering  alternative 
sources). 

•  Budget  issues. 

•  Commission  reports  such  as  the  Space 
Commission  [4],  or  reports  of  the 
Defense  Science  Board. 

•  Validity  of  your  organization’s  busi¬ 


ness  assumptions. 

•  Self-inflicted  pain. 

Many  organizations  do  not  have  a  clear 
understanding  of  who  are  their  custom¬ 
ers,  partners,  and  stakeholders.  Conse¬ 
quently,  it  is  difficult  to  know  underlying 
causes  for  dissatisfaction  and  key  issues 
that  need  to  be  addressed.  In  this  article, 
the  term  customers  refers  to  stakeholders, 
partners,  service  or  product  users,  and 
operators. 

Dissatisfaction  occurs  at  many  levels 
and  for  a  variety  of  reasons.  Sometimes  an 
organization  thinks  that  everything  is  fine 
because  its  average  product  or  service 
quality  is  high,  but  people  do  not  experi¬ 
ence  the  average  -  they  experience  the 
variability.  For  example,  telephone  service 
is  expected  to  be  excellent.  If  a  builder 
accidentally  cuts  a  telephone  cable,  leaving 
thousands  of  users  without  service  for 
five  days,  the  telephone  company  is  going 
to  have  a  lot  of  unhappy  customers. 
Customers  might  understand  a  one-day 
outage,  but  rapidly  lose  patience  with 
longer  periods.  In  areas  where  alternative 
service  is  provided  by  cable  television 
companies,  the  telephone  company  can 
lose  substantial  business  through  no  fault 
of  its  own. 

Determining  customer  satisfaction  as  a 
predictive  measure  is  very  difficult. 
Organizations  need  to  conduct  surveys, 
talk  to  customers,  and  collect  data.  A  com¬ 
pany’s  sales  are  a  backward-looking  indica¬ 
tor  of  customer  satisfaction.  When  people 
stop  buying  the  product,  it  is  frequently 
too  late  to  regain  market  share.  The  cur¬ 
rent  state  of  the  American  auto  industry  is 
a  classic  example  of  the  result  of  cus¬ 
tomer  dissatisfaction. 

Sometimes  customers  are  dissatisfied 
because  they  do  not  fully  understand  the 
services  provided.  Other  times,  dissatis¬ 
faction  might  be  related  to  budget  issues. 
Each  organization  needs  to  approach  cus¬ 
tomer  satisfaction  from  a  variety  of  per¬ 
spectives. 

Typical  budget  issues  include  not 
understanding  the  cost  of  doing  business, 
and  being  unable  to  defend  an  organiza¬ 
tion’s  budget  to  Congress.  The  latter  issue 
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is  related  to  stakeholders  not  understand¬ 
ing  all  the  services  provided  and  the  value 
of  these  services. 

Being  able  to  demonstrate  that  an 
organization’s  processes  are  lean  can  go  a 
long  way  toward  defending  its  budget.  It 
enables  the  organization  to  specify  the  ser¬ 
vices  that  will  be  lost  as  a  function  of  bud¬ 
get  reduction. 

This  concept  also  applies  to  acquisi¬ 
tion.  For  example,  if  an  organization  were 
to  acquire  the  next-generation  weather 
satellite  and  experienced  a  significant  bud¬ 
get  cut,  the  organization  should  be  able  to 
tell  Congress  of  the  impact.  The  organiza¬ 
tion  should  be  able  to  specify  how  this 
reduction  affects  the  products  that  can  be 
produced,  and  how  these  products  are 
related  to  weather  phenomena,  agribusi¬ 
ness,  transportation,  and  energy. 

Reasons  for  budget  reductions  include 
cost  overruns  by  major  programs  (in  other 
parts  of  the  agency  or  a  related  agency) 
and  pressures  to  balance  a  budget.  The  lat¬ 
ter  is  especially  prevalent  in  years  with 
major  national  catastrophes  such  as 
Hurricane  Katrina  and  years  in  which  wars 
are  being  fought. 

Sometimes  the  urgency  for  change  is 
provided  by  independent  commission 
reports.  For  example,  the  Space  Commis¬ 
sion  reported  on  significant  problems 
associated  with  space  programs  and  made 
a  number  of  suggestions  related  to  U.S. 
Air  Force  space  programs  and  the 
National  Reconnaissance  Office  (NRO). 
This  report  stated  that  the  NRO  has  lost 
its  edge.  The  chairman  of  this  committee, 
Donald  Rumsfeld,  needed  to  resign  a  little 
more  than  one  week  before  the  report  was 
delivered  to  the  secretary  of  defense 
because  he  was  to  be  sworn  in  as  the  sec¬ 
retary  of  defense.  Needless  to  say,  changes 
have  occurred  as  a  result  of  this  report. 

Space  Commission  reports  and  other 
congressional  actions  are  indicators  that 
the  organization’s  business  environment 
has  changed  —  that  its  business  assump¬ 
tions  may  no  longer  be  valid.  There  are 
many  examples  of  organizations  —  some¬ 
times  whole  industries  -  that  fail  to  under¬ 
stand  the  changes  to  their  environment. 
The  result  is  frequently  obsolete  or  incor¬ 
rect  business  models.  Examples  include 
the  following: 

•  The  failure  of  General  Motors  (GM) 
to  recognize  competition  from  Japan 
(and  other  issues). 

•  The  bankruptcy  or  near  bankruptcy  of 
all  major  airlines  because  they  use  a 
hub-and- spoke  system  and  have  very 
high  wages.  These  airlines  simply  can¬ 
not  compete  with  the  business  model 
of  discount  airlines  represented  by  Jet 


Blue  and  Southwest. 

•  Full-service  stock  brokerage  firms 
have  lost  substantial  business  to  dis¬ 
count  firms  and  online  trading. 

•  America  Online  (AOL)  is  still  trying  to 
understand  what  its  business  should  be 
in  the  light  of  high-speed  modems  and 
excellent,  free  search  engines.  Google 
recently  purchased  10  percent  of  AOL’s 
stock  and  could  become  the  driver  for 
defining  the  AOL  business  model. 

•  Wang  computers  had  a  virtual  strangle¬ 
hold  on  word  processing  and  office 
automation  in  the  1970s.  They  insisted 
on  maintaining  a  proprietary  system 
and  could  not  survive  the  competition 
of  Microsoft  Word  and  Word  Perfect. 
The  final  cause  for  urgency  of  change 

discussed  here  is  self-inflicted  pain,  which 
comes  from  a  variety  of  sources.  A  prima¬ 
ry  source  of  self-inflicted  pain  is  making  it 
difficult  to  conduct  business  with  your 
organization.  If  your  organization  pro¬ 
vides  services,  how  does  it  compare  with 
the  expected  service  that  people  receive 

“Without  fully 
committed  leaders , 
any  process 

improvement  or  change 
initiative  will  fail.  It  is 
simply  not  worth 
proceeding  if  you  do 
not  have  this  type  of 
commitment.^ 

from  national  companies  like  Nieman 
Marcus  or  Marriott?  Even  though  the 
business  areas  are  different,  your  organiza¬ 
tion  can  be  held  to  that  standard  because 
your  customers  and  stakeholders  might 
patronize  these  companies. 

Sometimes,  self-inflicted  pain  arises 
from  customers  and  stakeholders  not 
understanding  the  services  provided. 
These  situations  might  be  resolved 
through  a  good  education  or  marketing 
effort.  Other  sources  of  self-inflicted  pain 
are  derived  from  leadership  actions.  For 
example,  in  one  report  a  Navy  captain  ran 
a  guided  missile  cruiser  over  a  large  sub¬ 
merged  rock,  injuring  several  crew  mem¬ 
bers.  Sailors  on  the  bridge  suspected  the 
ship  was  headed  for  the  rock,  but  they 
were  afraid  to  tell  the  captain  to  change 


course  for  fear  of  being  wrong  [4]. 

As  you  look  at  the  urgency  for  change, 
tie  these  various  causes  together.  Would 
the  government  be  better  off  outsourcing 
your  services?  The  answer  is  usually  no, 
but  it  needs  to  be  supported  with  data. 

Facilitating  Change 

In  a  related  article,  the  co-author  and  I 
stated,  “The  three  most  important 
approaches  to  changing  culture  are  com¬ 
munications,  communications,  communi¬ 
cations”  [5].  My  new  and  improved  list  con¬ 
sists  of  12  items,  but  the  first  nine  are 
heavily  intertwined: 

1 .  Leadership. 

2.  Leadership. 

3.  Leadership. 

4.  Commitment. 

5.  Commitment. 

6.  Commitment. 

7.  Communications. 

8.  Communications. 

9.  Communications 

10.  Strategic  thinking. 

1 1 .  Consistency. 

12.  Understanding  the  data. 

Without  fully  committed  leaders,  any 
process  improvement  or  change  initiative 
will  fail.  It  is  simply  not  worth  proceeding 
if  you  do  not  have  this  type  of  commit¬ 
ment.  The  leader  of  a  small  group  might 
be  able  to  improve  the  processes  used  by 
his  or  her  group,  but  a  sustained  major 
change  requires  commitment  at  the  top. 

One  day  in  1993,  I  bumped  into  an 
acquaintance  who  was  president  and  chief 
operating  officer  of  a  major  corporation. 
He  said  that  he  was  leaving  the  next  day  to 
attend  a  weeklong  conference  on  business 
process  reengineering  (BPR).  Think  about 
the  cost  of  attending  that  conference. 
That  week  corresponds  to  2  percent  of  his 
time  for  the  entire  year.  Think  about  all  of 
the  other  items  commanding  his  attention 
that  will  not  be  done  during  that  week. 
Surely  every  vice  president  of  that  compa¬ 
ny  knew  that  he  was  attending  the  confer¬ 
ence,  and  understood  how  seriously  he 
was  taking  BPR. 

Somewhat  over  a  year  later,  I  asked  this 
person  about  his  accomplishments.  The 
results  were  somewhat  amazing  —  he  com¬ 
pletely  reengineered  a  major  production 
process  (investing  about  $125  million)  and 
entered  two  new,  but  related,  businesses. 
Success  was  achieved  because  of  leader¬ 
ship  and  commitment  at  the  top. 

Communication  is  also  critical  to  suc¬ 
cess.  To  enact  change,  it  is  necessary  to 
talk  to  everyone  in  the  organization  and  con¬ 
vey  the  urgency  for  change.  It  is  also  nec¬ 
essary  to  listen  to  people  and  to  under¬ 
stand  their  concerns.  These  concerns  can 
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be  somewhat  relieved  by  telling  a  story 
about  the  future.  Describe  your  vision  and 
why  it  is  necessary  The  Declaration  of 
Independence  tells  the  story  of  why 
change  was  necessary  The  U.S.  Consti¬ 
tution  is  a  formal  story  of  the  political 
utopia  envisioned  by  our  founding  fathers 
[6].  These  stories  tell  the  urgency  for 
change  and  how  people  will  be  affected  by 
the  change. 

Resistance  is  normal.  Recipients  of  bad 
news  go  through  a  series  of  psychological 
states:  denial,  anger,  bargaining,  despair, 
and  acceptance.  In  the  final  state  they  can 
act.  Change  leaders  need  to  understand 
resistance  to  change  and  to  communicate 
with  people  in  ways  that  mitigate  their 
resistance.  Much  of  this  communication 
involves  explaining  to  people  how  they  will 
be  affected  by  the  change. 

Gen.  Gordon  Sullivan  (retired)  dis¬ 
cusses  leadership  resistance  in  terms  of 
three  traps:  doing  things  too  well,  being  in 
the  wrong  business,  and  making  yesterday 
perfect  [7]. 

When  a  company  is  being  run  well,  it  is 
hard  to  recognize  the  need  for  change. 
Before  Jack  Welch  became  chief  executive 
officer  (CEO)  of  General  Electric  (GE), 
his  predecessor  was  viewed  as  the  top 
CEO  in  the  country.  Every  management 
school  viewed  GE  as  an  example  because 
they  were  highly  diversified,  immune  to 
the  ups  and  downs  of  business  cycles,  and 
regularly  grew  between  2  percent  and  3 
percent  per  year.  It  was  as  good  as  it  gets. 

Welch  recognized  the  need  for  change. 
He  immediately  said  that  GE  performance 
was  horrid.  GE  needed  to  grow  at  double¬ 
digit  rates.  GE  went  through  a  dramatic 
change,  going  from  80  percent  manufac¬ 
turing  and  20  percent  services  in  1982  to 
80  percent  services  and  20  percent  manu¬ 
facturing  in  2002.  During  this  20-year  peri¬ 
od,  the  company’s  stock  price  soared. 

Being  in  the  wrong  business  consists  of 
not  understanding  the  implications  of 
change.  Alfred  Sloan  said  that  GM  was  in 
the  business  of  making  money,  not  cars.  He 
also  said  that  success  comes  not  from  tech¬ 
nological  leadership,  but  from  having  the 
resources  to  quickly  adopt  the  innovations 
successfully  introduced  by  others  [8]. 
Consider  the  situation  of  GM  today  -  their 
bond  rating  has  been  reduced  to  junk. 

Making  yesterday  perfect  causes  peo¬ 
ple  to  ignore  or  rationalize  data  about  their 
environment.  In  military  terms,  making 
yesterday  perfect  corresponds  to  fighting 
yesterday’s  war.  The  entire  thrust  of  mov¬ 
ing  the  U.S.  Army  into  the  digital  age  —  the 
item  that  occupied  most  of  Sullivan’s  time 
when  he  was  chief  of  staff  —  has  been  to 
avoid  future  failure  by  being  prepared  for 


future  wars. 

Summarizing  this  section,  process 
improvement  involves  change.  Leaders 
must  be  committed  to  change  and  be  able 
to  communicate  the  urgency  for  change  to 
everybody  in  their  organization.  They 
need  to  listen,  think  strategically,  have  a 
reward  system  that  is  consistent  with  their 
goals,  and  understand  the  environmental 
data  to  which  they  are  exposed. 

Strategic  Thinking 

Strategic  thinking,  which  in  a  sense 
encompasses  leadership,  commitment, 
and  communications,  is  best  described  in 
terms  of  the  trigraph  depicted  in  Figure  l1. 
In  terms  of  change,  the  order  of  strategic 
thinking  is  strategy,  structure,  and  culture. 
Culture  and  structure  are  briefly  dis¬ 
cussed.  Most  of  the  discussion  in  this  sec¬ 
tion  focuses  on  strategy  because  that 
comes  first. 

There  are  a  variety  of  definitions  for 
culture,  but  most  of  them  are  related  to 
growth  or  transmitted  behavior.  At  the 
workplace,  culture  is  embodied  in  the 
rules  taught  to  the  next  generation  of 
workers.  Understanding  changes  to  these 
rules  for  success  takes  a  long  time.  People 
need  to  observe  whether  the  rewards  are 
actually  in  accordance  with  the  communi¬ 
cated  changes.  They  also  need  to  observe 
changes  in  the  organization’s  structure. 

It  is  reasonable  to  expect  culture 
change  to  take  from  five  to  seven  years  to 
be  fully  realized.  Culture  change  depends 
on  changes  to  the  organizational  structure 
and  to  the  strategy.  The  amount  of  time  it 
takes  for  change  cited  here  is  based  on  the 
author’s  observations  and  general  reading. 
The  author  is  unaware  of  specific  studies 
that  identify  time  required  for  change. 

The  most  important  step  for  changing 
culture  is  to  walk  the  talk  (pardon  the 
pun).  Employees  always  know  the  truth 
because  they  observe  leaders’  actions.  In 
1989,  a  survey  showed  that  43  percent  of 


employees  believed  that  management  lies 
and  cheats.  Another  survey  taken  in  1992 
showed  that  64  percent  of  the  employees 
think  that  management  lies  [9]. 

The  organization’s  actual  values  are 
communicated  through  actions  —  not 
posters.  For  example,  does  an  organiza¬ 
tion  reward  firefighters  or  fire  marshals?  If 
firefighters  are  rewarded,  do  not  expect 
the  employees  to  buy  into  process 
improvement  that  stresses  getting  the  job 
done  right  the  first  time. 

The  structure  of  an  organization  con¬ 
sists  of  organizational  charts  (i.e.,  report¬ 
ing),  roles,  and  relationships.  During  the 
first  year  of  change,  an  organization  gen¬ 
erally  focuses  on  strategy  —  values,  goals, 
and  vision.  Aligning  an  organization  with 
its  strategy  generally  takes  one  to  two 
additional  years.  Leaders  need  to  think 
about  where  they  want  to  be  in  10  years. 
They  then  need  to  determine  the  organi¬ 
zational  structure  that  is  needed  to  achieve 
their  goals  and  vision. 

The  structure  of  the  organization  usu¬ 
ally  needs  to  be  modified  so  that  the  right 
people  are  doing  the  right  jobs.  Sometimes, 
this  is  a  matter  of  reassigning  the  right  per¬ 
son  who  has  been  in  the  wrong  job.  Other 
times  it  is  necessary  to  bring  in  new  people, 
either  through  promotion  or  from  outside 
the  organization.  This  is  the  tough  part  of 
changing  the  structure. 

Using  GE  as  an  example  of  a  company 
that  has  undergone  major  cultural  changes 
over  the  years,  it  is  interesting  to  observe 
that  GE  has  never  brought  in  a  new  CEO 
from  the  outside.  Jack  Welch  was  a  lifelong 
GE  employee.  His  replacement,  Jeff 
Immelt,  was  promoted  from  within. 

Change  can  occur  from  within  an  orga¬ 
nization  by  having  the  right  people  in  the 
right  positions.  Jack  Welch  used  LSS  as  a 
tool  to  help  him  create  cultural  change.  At 
GE’s  1999  annual  meeting,  he  said  that  cor¬ 
porate  strategies  for  the  foreseeable  future 
were  globalization,  services,  and  Six  Sigma. 


Figure  1 :  The  Strategic  Thinking  Trigraph 


April  2006 


www.stsc.hill.af.mil  I  1 


Alternate  Mixes  for  CM  Ml 


If  Six  Sigma  or  CMM  IntegrationSM  is 
part  of  the  strategy,  how  is  this  to  be  done 
from  an  organizational  perspective?  Is  it 
part  of  a  corporate  quality  group?  Does 
each  division  need  to  have  a  process  czar? 
Even  when  we  know  what  needs  to  be 
done,  it  is  difficult  to  enact  these  changes 
in  industry,  and  much  more  difficult  in 
government.  It  frequently  requires  leaders 
to  act  in  the  best  interest  of  the  person 
who  will  come  after  them.  For  example, 
VADM  Phillip  Balisle  decided  to  imple¬ 
ment  LSS  throughout  the  Naval  Sea 
Systems  Command.  Not  long  after  Balisle 
made  this  decision,  VADM  Paul  Sullivan 
replaced  him.  Sullivan  has  also  committed 
to  LSS  and  is  implementing  it  throughout 
his  organization. 

First- Year  Strategy 

The  strategy  of  the  organization  is  gener¬ 
ally  defined  in  terms  of  its  mission,  core 
values,  vision,  and  goals.  Organizational 
leaders  are  generally  pretty  good  at  think¬ 
ing  in  these  terms;  however,  getting  the 
entire  senior  leadership  team  to  have  a 
common  understanding  generally  takes 
about  a  year. 

Holding  a  series  of  off-site  meetings 
with  advance  reading  and  homework  is  a 
primary  process  for  achieving  this  under¬ 
standing.  These  off-site  meetings  need  to 
be  reinforced  through  time  dedicated  to 
strategic  thinking  at  regular  staff  meetings. 
The  first  off-site  meeting  should  focus  on 
establishing  a  common  understanding  of 
the  organization’s  mission  and  values.  The 
mission  is  usually  understood  in  govern¬ 
ment.  Values  are  usually  only  implicitly 
known.  Each  leader  thinks  that  he  or  she 
knows  them,  and  in  most  good  organiza¬ 
tions  there  is  considerable  overlap  in  what 
people  think.  However,  this  off-site  meet¬ 
ing  is  likely  to  reveal  significant  differ¬ 
ences.  These  differences  form  the  basis 
for  extensive  discussion  of  the  organiza¬ 
tion’s  true  values. 

A  couple  of  good  reading  assignments 
to  be  completed  before  the  start  of  this  off¬ 
site  meeting  are  chapters  three  through  five 
(on  leadership  and  values)  of  Sullivan  and 
Harper  [10]  and  Larkin  and  Larkin  [11]. 

Other  topics  to  be  covered  in  this  first 
off- site  meeting  include  the  urgency  for 
change  and  a  brutally  honest  discussion  of 
the  organization’s  strengths  and  weakness¬ 
es.  For  example,  discuss  the  reasons  for 
recent  successes  and  failures.  Perform  a 
strengths,  weaknesses,  opportunities,  and 
threats  analysis. 

Discuss  the  reading  assignments.  Have 
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everybody  do  an  exercise  to  specify  why 
they  agree  or  disagree  with  Gen.  Sullivan’s 
rule  No.  2,  “Leadership  begins  with  val¬ 
ues.”  Include  discussion  of  leadership 
styles  as  part  of  this  session.  How  do 
these  styles  reflect  the  organization’s  val¬ 
ues?  Other  questions  to  ask  include: 

•  How  do  these  values  speak  to  the 
future  of  the  organization? 

•  How  does  organizational  and  individ¬ 
ual  behavior  reflect  these  values? 

•  What  behaviors  are  contradictory  to 
these  values? 

Subsequent  sessions  can  focus  on 
vision.  A  leader’s  vision  is  a  view  of  what 
his  or  her  organization  and  products  (or 
services,  etc.)  will  look  like  in  10  years.  It  is 
critical  that  the  vision  and  core  values  be 
in  harmony.  Each  member  of  the  senior 
leadership  team  should  write  a  story  that 


“Leaders  must  be 
committed  to  the  change 
and  be  able  to 
communicate  the 
urgency  for  change  to 
everybody  in  their 
organization.,> 

describes  the  vision  in  terms  of  his  or  her 
business  area  or  function.  Telling  stories  is 
a  great  way  to  learn  [12]. 

Core  values  and  vision  act  like  fields  — 
electric,  magnetic,  gravitational,  etc.  They 
are  unseen,  but  are  real  forces  reflected 
throughout  the  organization.  (See  [13]  for 
further  discussion  of  this  and  related  con¬ 
cepts.) 

Consider  the  Tylenol  scare  of  1982 
and  Johnson  &  Johnson’s  reaction.  Seven 
people  died  on  Chicago’s  west  side  when 
they  ingested  extra- strength  Tylenol  cap¬ 
sules  laced  with  cyanide.  Even  though  this 
problem  was  not  caused  by  Johnson  & 
Johnson,  they  instantly  pulled  the  product 
from  the  market  and  kept  it  off  the  mar¬ 
ket  until  tamper-proof  packaging  (and 
caplets)  was  developed.  It  cost  the  compa¬ 
ny  more  than  $100  million. 

When  asked  how  the  company  was 
able  to  respond  so  quickly,  the  CEO  of 
Johnson  &  Johnson  said  that  not  doing  so 
would  have  violated  its  core  values.  He  did 
not  need  to  call  meetings;  he  just  applied 
their  core  values  [14]. 

The  CEO’s  actions  were  highly  visible 
to  both  employees  and  the  public.  Every 


employee  in  this  global  company  knew  what 
happened.  Within  three  months,  Tylenol 
regained  95  percent  of  its  market  share. 

Communicate  your  values  and  vision 
to  everyone  in  your  organization. 

The  senior  leadership  team  also  needs 
to  define  goals,  activities,  and  metrics. 
Goals  are  long-term  and  when  achieved, 
an  organization’s  vision  will  be  substantial¬ 
ly  realized.  Activities  are  near-term  tasks 
such  as  plans  for  the  next  fiscal  year  and 
are  tied  to  the  goals.  Metrics  measure 
progress  toward  achieving  the  goals. 

LSS  is  an  example  of  an  activity. 
Depending  on  the  goals,  it  could  be  aimed 
toward  making  the  organization  world- 
class,  reducing  costs  to  meet  future  budgets, 
or  providing  value  to  customers.  Whatever 
its  aim,  justify  it  by  measuring  results. 

Summary 

This  article  focuses  on  the  leadership 
actions  needed  to  implement  process 
improvement,  which  involves  changing  an 
organization’s  culture.  That  topic  is  gener¬ 
ally  not  discussed  when  we  espouse  CMM, 
LSS,  or  any  other  process  improvement 
method  yet  it  is  critical  to  successful 
implementation.  This  attempted  culture 
change  will  cause  the  ultimate  demise  of 
process  improvement  efforts  unless  it  is 
handled  well. 

For  those  reasons,  leadership  is  the 
focus  of  this  article.  It  is  pointed  out  that 
change  is  hard.  Successful  change  requires 
committed  leaders  who  communicate  with 
their  employees  and  have  action  consistent 
with  what  they  say.  Leaders  need  to  do  a  lot 
of  strategic  thinking.  They  need  to  recog¬ 
nize  that  change  occurs  over  time.  Patience 
is  necessary.  Leaders  need  to  establish  their 
strategy  during  the  first  year  and  develop 
an  appropriate  organizational  structure 
within  three  years.  Then,  through  consis¬ 
tency,  paying  attention  to  the  data,  and 
communication,  the  culture  will  change. 

This  approach  is  encapsulated  in  the 
title  of  Peter  Schwartz’s  book  “The  Art  of 
the  Long  View.”^ 
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Note 

1.  The  original  source  of  this  figure  is 
uncertain,  but  I  was  first  shown  it  by 
Sheila  Sheinberg,  a  nationally  known 
consultant  in  organizational  behavior. 
Some  of  the  ideas  in  this  article  come 
from  working  with  her. 
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risk  evaluation.  Though  risk  evaluations  are  only  one  component  of  a  source  selection,  all  bidders  underwent  the  risk  evalu¬ 
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sentatives  from  the  government  and  the  organisation  being  appraised  work  together  on  the  appraisal  team  to  jointly  evaluate 
the  organisation.  This  article  describes  the  efforts  of  the  authors  in  performing  post-award  cooperative  appraisals,  the  lessons 
learned,  and  the  benefits  to  both  the  government  and  the  appraised  organisation. 


Traditionally,  solicitations  for  Depart¬ 
ment  of  Defense  (DoD)  projects 
have  included  some  method  of  risk  evalu¬ 
ation  to  determine  the  level  of  risk  that 
the  project  manager  (PM)  will  face  in 
selecting  a  bidder  to  provide  the  prod¬ 
ucts/  services  for  his  or  her  program.  This 
risk  evaluation  could  take  the  form  of  a 
Software  Capability  Evaluation  (SCESM),  a 
methodology  developed  by  the  Software 
Engineering  Institute  (SEISM)  at  Carnegie 
Mellon  University,  or  other  methods. 

The  Communications-Electronics  Life 
Cycle  Management  Command  (C-E 
LCMC)  Software  Engineering  Center 
(SEC)  developed  a  streamlined  form  of 
the  evaluation  called  the  Software  Process 
Risk  Evaluation  (SPRE)  that  has  been 
mandated  for  all  major  C-E  LCMC  acqui¬ 
sitions.  This  method,  like  the  SCE,  has 
been  used  during  the  solicitation  process 
to  evaluate  all  potential  vendors  and  pro¬ 
vide  input  to  the  evaluation  factors  for  the 
solicitation. 

There  were  several  problems  with  the 
SCE  and  SPRE  methods.  There  often  was 
not  enough  time  to  prepare  for  the  evalu¬ 
ation.  A  lack  of  historical  information 
about  the  bidders’  processes  made  the 
evaluations  more  critical  while  decreasing 
the  effectiveness  of  a  short,  intense  on¬ 
site  visit.  The  cost  of  the  evaluations  were 
high  for  both  the  government,  who  has  to 
visit  all  of  the  bidders  during  the  propos¬ 
al  evaluation  period,  and  the  contractor, 
who  has  to  apply  significant  resources  to 
prepare  data  and  provide  people  to  be 
interviewed  during  the  on-site  visit.  Both 
parties  potentially  spend  money  to  sup¬ 
port  multiple  source  selection  evaluations. 
The  government  expends  resources  to 

SM  CMM  Integration,  SCE,  SCAMPI,  and  SEI  are  service 
marks  of  Carnegie  Mellon  University. 


evaluate  the  losing  contractors,  and  the 
contractor  could  possibly  have  to  support 
multiple  evaluations  in  a  given  timeframe. 

Acquisition  Reform 

Industry  has  matured  over  the  years.  The 
SEI  issued  the  Capability  Maturity 
Model®  for  Software  (SW-CMM®)  in 
1993.  Since  that  time,  many  of  the  orga¬ 
nizations  that  bid  on  acquisition  solicita¬ 
tions  have  undergone  process  improve¬ 
ment  initiatives  using  SW-CMM  or  its 
successors,  including  the  CMM  Inte¬ 
gration^  (CMMI®).  With  the  advent  of 
the  CMMI,  a  new  appraisal  method  -  the 
Standard  CMMI  Appraisal  Method  for 
Process  Improvement  (SCAMPISM)  [1]  - 
has  been  developed  and  is  quickly  becom¬ 
ing  the  appraisal  method  of  choice, 
regardless  of  which  CMM  is  being  used 
as  the  model  for  process  improvement. 
SCAMPI  can  be  used  in  lieu  of  the 
SCE/SPRE  for  evaluating  bidders  during 
a  source  selection. 

Government  acquisition  reform  has 
evolved  as  well.  The  Interim  Defense 
Acquisition  Guidebook,  paragraph 
C5. 2. 3. 5. 6. 1.5,  states  the  following: 

Select  contractors  with  domain 
experience  in  developing  compara¬ 
ble  software  systems;  with  success¬ 
ful  past  performance;  and  with  a 
mature  software  development  capa¬ 
bility  and  process.  Contractors  per¬ 
forming  software  development  or 
upgrade (s)  for  use  in  an  [Acqui¬ 
sition  Category]  ACAT  I  or  ACAT 
IA  program  shall  undergo  an  evalu¬ 
ation,  using  either  the  tools  devel¬ 
oped  by  the  [SEI]  or  those 
approved  by  both  the  DoD 
Components  and  the  Deputy 
Director,  Software  Intensive  Sys¬ 


tems.  At  a  minimum,  full  compli¬ 
ance  with  SEI  Capability  Maturity 
Model  Level  3,  or  its  equivalent  in 
an  approved  evaluation  tool,  is  the 
department’s  goal.  However,  if  the 
prospective  contractor  does  not 
meet  full  compliance,  risk  mitiga¬ 
tion  planning  shall  describe,  in 
detail,  the  schedule  and  actions  that 
will  be  taken  to  remove  deficiencies 
uncovered  in  the  evaluation  process. 
Risk  mitigation  planning  shall 
require  Product  Manager  approval. 
The  Deputy  Director,  Software 
Intensive  Systems  shall  define  Level 
3  equivalence  for  approved  evalua¬ 
tion  tools.  The  evaluation  shall 
examine  the  business  unit  proposed 
to  perform  the  work.  The  reuse  of 
existing  evaluation  results  per¬ 
formed  within  a  2-year  period  prior 
to  the  date  of  the  government  solic¬ 
itation  is  encouraged.  [2] 

Later  guidance  clarified  Level  3  equiv¬ 
alence.  The  Software  Development 
Capability  Evaluation  [3],  developed  by 
the  U.S.  Air  Force  with  approved  core  set 
revisions,  was  determined  to  be  an  accept¬ 
able  alternative  set  of  criteria  to  the  SW- 
CMM.  The  use  of  CMMI  Systems 
Engineering  and  Software  Engineering 
Vers.  1.1  Level  3  criteria  is  another  accept¬ 
able  alternative  to  the  use  of  SW-CMM 
Level  3  criteria. 

Newer  guidance  from  the  Office  of 
the  Secretary  of  Defense/Acquisition, 
Technology,  and  Logistics  no  longer 
requires  Level  3  equivalent  ratings. 
Process  improvement  is  encouraged,  but 
maturity  levels  are  no  longer  stated  in  the 
guidance.  The  rationale  is  that  many 
organizations  have  reached  maturity 
Level  3  or  better,  and  that  the  original 
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direction  was  not  to  have  organizations 
go  for  the  rating,  but  to  initiate  organiza¬ 
tional  process  improvement. 

The  CMMI,  in  its  continuous  represen¬ 
tation,  allows  an  organization  to  be 
appraised  to  determine  a  process  capabili¬ 
ty  profile;  the  organization  can  look  at 
Process  Areas  (PAs)  of  interest  based  on 
organizational  goals  and  see  how  well  they 
have  implemented  them.  This  can  also 
allow  a  PM  to  focus  on  areas  that  are  of 
high  risk  for  a  particular  acquisition,  rather 
than  look  at  overall  maturity  across  all  PAs. 

Appraisal  Reuse 

To  reduce  costs  and  resources  associated 
with  evaluating  development  capabilities, 
the  DoD  is  working  with  industry  to  pro¬ 
vide  implementing  mechanisms  that  better 
support  the  reuse  of  appraisals.  Results  of 
previous  appraisals  conducted  on  the  orga¬ 
nizational  unit  proposing  to  do  the  work 
may  be  an  acceptable  alternative  to  the 
government  performing  a  new  appraisal. 
The  reused  appraisals  must  be  shown  to  be 
independent,  i.e.,  at  a  minimum,  the  lead 
appraiser  should  not  be  from  within  the 
organization  being  appraised. 

C-E  LCMC  Acquisition  Strategy 

The  SEC  has  been  involved  in  the  exami¬ 
nation  of  cooperative  government /indus¬ 
try  appraisals  as  an  alternative  to  the  con¬ 
ventional  acquisition  strategy  for  several 
years  [4].  The  SEC  took  the  initiative  in 
working  with  the  acquisition  center  at 
Fort  Monmouth  to  identify  software¬ 
intensive  acquisitions  occurring  there. 
The  interim  project  office  was  contacted 
and  cost-effective  strategies  developed  to 
address  the  acquisition  agent’s  need  to 
reduce  risk.  One  of  these  strategies  was  to 
wait  until  after  contract  award  and  per¬ 
form  a  post-award  appraisal  of  the  suc¬ 
cessful  bidder. 

To  make  the  post-award  appraisal 
viable,  the  solicitation  package  must  have 
the  appropriate  language.  The  contract 
should  require  that  the  product  be  devel¬ 
oped  using  CMM/CMMI  Level  3 
processes.  Proof  is  to  be  submitted, 
demonstrating  that  the  contractor  is  rated 
as  a  Level  3  organization  (e.g.,  copies  of 
CMM/CMMI  appraisals  and  process 
improvement  track  record).  If  the  organi¬ 
zation  cannot  verify  Level  3,  a  detailed 
process  improvement  plan,  including  a 
schedule  that  leads  to  a  Level  3  appraisal 
is  submitted.  This  provides  the  acquisi¬ 
tion  agent  with  material  in  which  to  eval¬ 
uate  the  bidders’  process  maturity  during 
the  source  selection  without  the  effort 
required  for  on-site  visits. 

The  post-award  appraisal  must  also  be 


contained  in  the  contract.  Typically,  C-E 
LCMC  requires  that  the  contractor  is 
responsible  for  leading  and  conducting  an 
appraisal  with  government  participation 
on  the  appraisal  team.  This  is  appropriate 
as  it  is  the  contractor’s  plant  and  process¬ 
es.  The  contractor  is  required  to  submit  an 
appraisal  plan  or  process  proposal  as  a 
formal  deliverable  after  the  contract  is 
awarded. 

If  a  CMMI  SCAMPI  appraisal  is 
selected  or  required,  additional  options 
present  themselves.  The  SCAMPI  frame¬ 
work  defines  three  classes  of  appraisal.  A 
Class  C  appraisal  is  a  very  cursory  look  at 
the  processes  of  the  organization.  It  can 
often  be  nothing  more  than  a  review  of 
process  documentation  and  its  application 
to  the  project.  A  Class  B  appraisal  is  more 


“The  CMMI ,  in 
its  continuous 
representation ,  allows 
an  organization  to  be 
appraised  to 
determine  a  process 
capability  profile ;  the 
organization  can  look  at 
Process  Areas  (PAs)  of 
interest  based  on 
organizational  goals  and 
see  how  well  they  have 
implemented  them.” 

robust,  but  does  not  emphasize  the  depth 
of  coverage  and  rigor  that  result  in  a 
maturity  level  rating  for  the  organization. 
Class  A  appraisals  are  performed  by  SEI- 
authorized  SCAMPI  lead  appraisers.  They 
can  lead  to  a  formal  maturity  rating  or 
capability  profile  that  is  submitted  to  the 
SEI.  The  government  determines  which 
class  SCAMPI  appraisal  is  appropriate  for 
the  acquisition. 

With  the  SEI  developing  formal  mech¬ 
anisms  for  SCAMPI  B  and  C  appraisals, 
this  methodology  is  fitting  into  a  DoD 
approach  suggested  by  Mark  Schaeffer, 
the  DoD  sponsor  for  CMMI,  and 
Director,  Systems  Engineering  for  the 


Office  of  the  Secretary  of  Defense.  This 
approach,  reported  by  Mike  Phillips  in 
“CMMI:  A  Progress  Report”  [5],  would 
have  acquirers  look  for  strengths  or  weak¬ 
nesses  in  the  development  processes  that 
constitute  risks  to  the  proposed  develop¬ 
ment  effort.  Satisfying  CMMI  goals, 
process  areas,  or  maturity  or  capability  lev¬ 
els  would  not  be  the  point;  it  is  primarily  a 
risk  identification  and  mitigation 
approach. 

Schaeffer  suggests  a  three-phased 
approach.  In  phase  1,  bidders  would  be 
appraised  for  PAs  that  the  Government 
Program  Office  considers  highest  risk. 
After  contract  award,  in  phase  2,  the  win¬ 
ning  team  undergoes  a  baseline  appraisal, 
using  a  risk-based  analysis  of  the  process 
strengths  and  weaknesses,  thereby  estab¬ 
lishing  action  plans  for  future  checks.  In  a 
risk-based  appraisal  framework,  the 
CMMI  is  used  to  identify  and  group  weak¬ 
nesses  to  address  systemic  problems.  The 
CMMI’s  process  categories  -  Project 
Management,  Engineering,  Support,  and 
Process  Management  -  may  be  used  to 
define  areas  that  the  PM  believes  most 
affects  the  developer  team’s  contract  per¬ 
formance.  Using  these  categories  effec¬ 
tively  covers  both  product  and  process 
risk.  In  phase  3,  the  risks  are  monitored  to 
closure. 

Tying  CMMI/risk  mitigation  to  an 
award  fee  on  a  contract  vehicle  can  be  a 
good  incentive  for  the  contractor.  A  pro¬ 
gram  office  should  consider  wording  the 
contract  to  have  a  continuous  process 
monitoring  function.  For  example,  a  PM 
could  elect  to  utilize  the  Defense  Contract 
Management  Agency  (DCMA)  or  some 
other  matrix  organization  to  perform 
process  monitoring.  This  is  also  a  recom¬ 
mendation  made  in  the  Workshop  on 
CMMI  Use  in  Acquisition  [6]. 

Benefits  of  Post-Award 
Appraisals 

The  PM  saves  resources  by  limiting  the 
number  of  appraisals  required  for  a  solic¬ 
itation.  Maturity  risk  is  only  one  of  many 
evaluation  factors  for  an  acquisition 
source  selection  evaluation.  The  increased 
risk  of  not  performing  an  on-site  evalua¬ 
tion  for  each  bid  is  mitigated  by  the  cost 
and  time  savings.  Many  organizations  have 
been  performing  process  improvement 
initiatives  for  more  than  10  years.  This 
legacy  of  process  improvement  tends  to 
ensure  that  processes  are  defined,  which  is 
the  focus  of  maturity  Level  3  in  the 
CMM/CMMI. 

The  bidders  benefit  in  this  approach  as 
well.  Organizations  can  reuse  their 
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Attribute 

SCE/SPRE/SCAMPI 

Cooperative  Appraisal 

Timeframe 

■  Pre-award. 

■  May  be  used  post-award  for 
contract  monitoring. 

■  Pre-award  or  post-award. 

■  Recommend  post-award 
baseline  and  follow-on  for 
contract  monitoring. 

■  Can  be  linked  to  supplier's 
process  improvement  appraisal 
schedule. 

Cost 

■  High  (both  supplier  and 
acquirer). 

■  Lower  (cost  sharing). 

Cost  Effectiveness 

■  Low,  especially  if  supplier  must 
support  evaluations  for  losing 
solicitation. 

■  High. 

■  Eliminates  cost  of  evaluating 
losing  bidders. 

Resources  Needed 

■  Resource-intensive. 

■  Resource  sharing  for  staffing 
appraisals. 

■  Supplier  can  reuse  process 
data. 

Incentive  for  Process 
Improvement 

■  Little  if  no  contractual  language 
for  process  improvement. 

■  Supplier  can  reuse  process 
assets  developed  for  process 
improvement. 

■  Potential  for  appraisal  reuse  on 
subsequent  solicitations. 

Integrated  Product 
Development 

■  Government-only  team  does  not 
form  integrated  teams  with 
supplier. 

■  Fosters  early  development  of 
integrated  teams. 

■  Facilitates  government/supplier 
communication. 

Risk 

■  Mitigates  selection  of  high-risk 
bidders. 

■  Does  not  allow  for  an  in  depth 
risk  evaluation  of  the  supplier 
unless  applied  post-award. 

■  Early  risk  mitigation  if  performed 
early  in  contract  execution. 

■  Continuous  risk  monitoring  and 
control  if  used  for  contract 
monitoring. 

Table  1 :  Evaluation  and  Cooperative  Appraisal  Attributes 


process  data  for  multiple  solicitations. 
They  do  not  have  the  expense  of  prepar¬ 
ing  for  an  on-site  visit  for  a  solicitation 
that  they  are  not  guaranteed  to  win. 
Cooperative  appraisals  reward  the  organi¬ 
zation  for  its  process  improvement 
efforts,  thereby  encouraging  internal 
process  improvement  activities.  Table  1 
summarizes  the  attributes  and  how  con¬ 
ventional  evaluations  compare/ contrast  to 
the  cooperative  appraisal  methodology. 

Pilot  Effort  for  Cooperative 
Appraisals 

The  SEC  was  able  to  participate  in  an 
ACAT  I  acquisition  where  the  PM  was 
consulted  before  the  release  of  the 
Statement  of  Work  (SOW).  After  being 
briefed  on  the  benefits  of  performing  a 
post- award  appraisal,  the  PM  agreed  to 
have  language  added  to  the  SOW  for  a 
CMMI  self-assessment  of  all  major  con¬ 
tributors  to  the  product  development 
effort  using  the  SCAMPI  method  with  up 
to  four  government  participants. 

The  winning  contractor  team  submit¬ 
ted  an  assessment  plan  to  have  four  team¬ 
mates  undergo  a  self-assessment.  The 
original  plan  called  out  in  the  SOW  was  to 
submit  the  plan  within  60  days  after  con¬ 
tract  award  and  perform  the  self-assess¬ 
ments  within  120  days  after  contract 
award.  The  assessment  would  establish  a 
process  capability  baseline  on  the  organi¬ 
zational  unit  doing  the  work  on  the  pro¬ 
gram  as  well  as  determine  risk  areas  to  be 


monitored  over  the  acquisition  life  cycle. 
Since  the  contract  was  awarded  to  a  team 
of  contractors,  consideration  was  given  to 
the  timing  requirements  for  both  the  plan 
and  the  conduct  of  the  self-assessments. 
The  SOW  provided  a  list  of  CMMI  PAs  of 
interest.  These  were  the  minimum  set  of 
processes  that  were  to  be  assessed,  which 
tied  into  the  program  office’s  key  perfor¬ 
mance  parameters,  goals,  and  objectives 
for  the  program. 

Using  multi-organizational  teams 
brings  a  new  risk  to  the  conventional 
acquisition  evaluation  methodology.  The 
contracting  team  may  have  individually 
performed  process  improvement  efforts, 
but  the  team  does  not  necessarily  share 
institutionalized  processes.  Former  assess¬ 
ments  are  not  directly  applicable  to  the 
newly  formed  team  and  are  less  predictive 
of  the  maturity  or  capability  of  the  team 
doing  the  work. 

In  this  pilot,  the  prime  contractor 
worked  to  develop  capstone  processes 
that  all  teammates  would  follow,  but  addi¬ 
tionally  allowed  for  the  individual  organi¬ 
zation’s  processes  to  be  used.  For  exam¬ 
ple,  a  project  Software  Development  Plan 
(SDP)  was  written  by  the  prime,  which 
called  out  use  of  the  other  organizations’ 
SDPs  for  their  software  contributions  to 
the  project. 

The  contractor’s  self-assessment  plan 
called  for  self-assessments  to  be  held  at 
multiple  organizations.  Each  self-assess¬ 
ment  covered  all  maturity  Level  2  and  3 


PAs.  Some  sites  added  higher  maturity 
PAs.  The  government  worked  with  the 
contractor  team  to  try  to  leverage  each 
organization’s  internal  process  improve¬ 
ment  activities.  All  of  the  contractors  were 
transitioning  from  SW-CMM  to  CMMI; 
they  were  already  implementing  organiza¬ 
tional  process  improvement  plans. 

Where  feasible,  the  government 
allowed  the  organization  to  add  the  acqui¬ 
sition  project  to  the  list  of  internal  pro¬ 
jects  being  appraised  to  avoid  unneces¬ 
sary  effort  of  preparing  for  an  indepen¬ 
dent  assessment  of  one  program.  For  at 
least  one  organization,  this  meant  sched¬ 
uling  the  appraisal  as  part  of  their  exter¬ 
nally  led  SCAMPI.  This  benefited  the 
government  as  well,  since  the  project  was 
not  yet  fully  under  way  and,  therefore, 
many  PAs  (e.g.,  technical  solution,  verifi¬ 
cation,  and  validation)  could  not  be  rigor¬ 
ously  assessed  —  the  project  simply  was 
not  to  that  point  in  its  life  cycle.  The  addi¬ 
tion  of  projects  in  different  stages  of 
maturity  provided  a  more  rounded  picture 
of  the  organization  and  how  its  institu¬ 
tionalized  processes  would  be  applied  to  a 
future  project. 

The  SEC  provided  the  two  authors  as 
the  core  team  that  participated  on  the  self- 
assessments.  Using  available  local  DCMA 
representatives  augmented  government 
participation.  The  PM  had  contracted 
with  DCMA  to  perform  process  monitor¬ 
ing  over  the  course  of  the  development 
effort,  so  their  participation  in  the  self- 
assessments  acted  as  a  kick-start  in  under¬ 
standing  the  details  of  the  project’s 
processes. 

Cooperative  Appraisals  and 
the  CMMI 

The  way  that  acquisition  reform  is  evolv¬ 
ing  is  similar  to  that  of  the  evolution  from 
SW-CMM  to  CMMI.  CMMI  provides 
integration  of  software  with  other  disci¬ 
plines,  i.e.,  systems  engineering  and  inte¬ 
grated  product  and  process  development 
(IPPD),  manifested  in  integrated  product 
teams  (IPTs). 

There  is  an  analogous  evolution  in 
acquisition  —  a  paradigm  shift  from  the 
traditional  buyer-seller  relationship  to  that 
of  an  IPT  where  both  the  government 
and  contractor  share  responsibility  for  the 
end  product.  This  integrated  team  con¬ 
cept  necessitates  a  team  approach  to 
assessment  as  well.  In  a  cooperative 
appraisal,  the  government  participants  add 
objectivity  and  diverse  experience.  The 
team  that  is  formed  during  cooperative 
appraisals  can  be  extended  to  established 
IPTs  for  the  acquisition  life  cycle,  where 
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government  participants  in  the  assessment 
can  also  be  members  of  the  government’s 
project  management  team. 

Anyone  who  has  participated  in  an 
SEI  appraisal  can  attest  to  the  fact  that 
the  intensity  of  the  shared  experience 
does  much  to  foster  team  building.  A 
foundation  of  mutual  respect,  equality, 
and  cooperation  is  established  through¬ 
out  an  appraisal.  These  sentiments  can  be 
brought  back  to  the  project  management 
office  and  used  to  continuously  facilitate 
communication  between  the  government 
and  contractor.  The  cooperative 
appraisal  process  allows  both  parties  to 
utilize  and  benefit  from  the  appraisal 
data;  there  is  open  process  communica¬ 
tion  between  the  government  and  their 
contractor.  This  gives  the  government  an 
understanding  of  the  way  the  contractor 
does  business. 

The  government  gets  to  meet  the  key 
players,  from  the  president  of  the  corpo¬ 
ration  to  the  practitioners  at  the  develop¬ 
er’s  site,  and  may  additionally  see  the  facil¬ 
ities  and  operations  for  the  project.  This 
interaction  establishes  positive  working 
relationships  and  provides  a  greater 
understanding  of  what  the  government/ 
contractor  IPT  brings  to  the  program. 

In  a  cooperative  appraisal,  the  govern¬ 
ment  leverages  the  project  and  process 
expertise  brought  by  the  contractor,  thus 
facilitating  the  government’s  assessment 
efforts.  Usually  there  are  appraisal  team 
members  that  are  part  of  the  contractor’s 
process  improvement  group  or  belong  to 
the  team  working  on  one  of  the  projects 
being  appraised.  These  people  can  quickly 
guide  the  government  appraisal  team  mem¬ 
ber  to  the  appropriate  process  artifacts  or 
answer  appraisal-focused  questions. 

Lessons  Learned 

There  was  some  trepidation  on  the  part  of 
the  contractors  in  performing  the  cooper¬ 
ative  appraisal.  It  was  initially  perceived 
that  the  government  representatives  were 
coming  in  to  perform  an  audit  or  evalua¬ 
tion,  not  to  participate  in  an  internal 
process  appraisal.  Discussions  with  the 
site  coordinator  and  contractor  manage¬ 
ment  before  the  appraisal  helped  over¬ 
come  these  perceptions. 

The  government  representatives 
explained  that  the  cooperative  appraisals 
were  one  more  manifestation  of  the  IPT 
method  of  running  a  program  where  the 
government  and  the  contractor  share 
responsibility  for  the  program’s  success. 
The  contractors  were  reminded  that  their 
organization’s  formal  appraisal  teams  are 
often  composed  of  personnel  external  to 
the  organization,  and  that  the  government 


representatives’  goal  was  to  be  integral 
members  of  the  team.  The  measure  of 
success  would  be  how  forthcoming  those 
interviewed  were  during  the  self-assess¬ 
ment.  Since  many  of  the  people  inter¬ 
viewed  had  experience  with  internal 
appraisals,  where  they  may  speak  with 
people  outside  of  their  business  unit  or 
external  people,  the  cooperative  appraisal 
team  did  not  experience  any  difficulties  in 
this  area. 

Internal  process  improvement 
requires  senior  management  commitment. 
The  cooperative  appraisal  teams  usually 
are  comprised  of  several  process  group 
members.  These  process  group  personnel 
are  responsible  for  recommending  the 
future  direction  of  process  improvement 
efforts  in  the  organization.  The  organiza¬ 
tions  being  appraised  were  transitioning 
from  CMM  to  CMMI;  some  had  never 
piloted  a  CMMI  appraisal.  This  provided 
an  opportunity  for  the  process  group  to 

"...  the  authors  can 
state  that  an  extremely 
beneficial  byproduct 
of  being  on  an 
appraisal  team  is 
that  you  can  learn 
of  many  practices 
that  can  be  adapted 
for  use  in 

your  organization” 

gather  valuable  information  on  the  state 
of  the  organization  and  how  successfully 
the  transition  efforts  were  proceeding. 
The  external  inputs  from  the  government 
can  assist  the  process  group  by  providing 
an  unbiased  view  of  the  organization,  and 
help  influence  senior  management  in 
determining  what  areas  the  organization 
should  be  concentrating  on  in  their  future 
process  improvement  initiatives. 

The  government  gained  insight  into 
the  different  cultures  of  the  organizations 
comprising  the  contractor  team.  They  had 
different  ways  of  doing  business,  different 
vocabularies,  and  different  ways  of  work¬ 
ing  with  the  prime  contractor  and  the 
government.  Understanding  these  differ¬ 
ences  aids  in  facilitating  communications 
among  the  members  of  the  IPTs  and 


avoids  misinterpretations. 

Having  a  core  team  of  government 
participants  on  all  of  the  appraisals  pro¬ 
vided  continuity.  The  same  personnel  can 
compare  and  contrast  how  the  different 
organizations  are  performing  process 
improvement  and  satisfying  the  practices 
of  the  CMMI.  Common  areas  of  weak¬ 
ness  and  interpretation  issues  can  be 
raised  and  addressed  to  better  support  the 
program. 

If  the  government  provides  several 
people  for  a  cooperative  appraisal,  they 
should  represent  the  major  disciplines 
(e.g.,  program  management,  systems  engi¬ 
neering,  software  engineering,  and  logis¬ 
tics)  that  are  involved  in  the  acquisition.  A 
multi-disciplinary  team  can  cover  a  broad¬ 
er  range  of  PAs,  while  a  given  government 
team  member  can  specialize  in  his  or  her 
area  of  expertise.  Involving  multiple  disci¬ 
plines  allows  the  government  to  examine 
the  developers’  processes,  taking  into  con¬ 
sideration  the  entire  program  life  cycle. 
This  enables  downstream  risks  to  be  iden¬ 
tified  early  in  the  acquisition.  These  risks 
can  be  mitigated  with  less  effort  as  a  result 
of  the  early  collaboration  and  expertise 
among  the  government  appraisal  team 
and  the  developers’  project  teams. 

PMs  considering  using  cooperative 
appraisals  should  enlist  direct  or  matrix 
support  from  the  PM  office  for  participa¬ 
tion  on  the  appraisal  team.  The  detailed 
view  of  the  organization  is  extremely 
valuable  to  a  person  who  is  supporting  the 
PM  in  managing  the  acquisition.  The 
appraisal  team  member  gets  to  meet  and 
speak  with  many  of  the  people  perform¬ 
ing  the  project  work,  interview  senior 
management,  and  understand  their  com¬ 
mitment  to  the  project. 

As  participants  of  several  appraisals, 
the  authors  can  state  that  an  extremely 
beneficial  byproduct  of  being  on  an 
appraisal  team  is  that  you  can  learn  of 
many  practices  that  can  be  adapted  for  use 
in  your  organization.  No  organization  has 
a  monopoly  on  best  practices.  The  more 
exposure  that  you  have  to  the  industry,  the 
more  you  can  recognize  there  are  superior 
ways  of  doing  things. 

Aftermath 

The  joint  appraisals  were  considered  to 
be  a  success  by  both  the  government  and 
industry.  Two  of  the  teammates  who  had 
been  appraised  invited  the  government 
representatives  back  for  a  follow-on 
appraisal  where  the  company  was  trying 
to  achieve  a  higher  maturity  level.  These 
organizations  valued  the  government’s 
contributions  to  the  appraisal  effort,  and 
wished  to  maintain  the  same  experienced 
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appraisal  team  in  this  follow-on  appraisal. 
The  government,  in  turn,  appreciated  the 
opportunity  to  follow  up  on  re-appraising 
the  organizations.  This  allowed  the  gov¬ 
ernment  the  ability  to  witness  the  effect  of 
the  process  improvements  made  in  the 
organizations  since  the  self-assessment.^ 
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Software  development  tools  are  in  wide  use  among  safety-critical  system  developers.  Examples  of  such  use  include  aviation,  auto¬ 
motive,  space,  nuclear,  railroad,  medical,  and  military  applications.  However,  verification  of  tool  output  to  ensure  safety,  man¬ 
dated  in  highly  regulated  industries,  requires  enormous  effort.  If  a  tool  is  qualified,  this  effort  can  be  reduced  or  even  eliminat¬ 
ed.  The  Rwidio  Technical  Commission for  Aeronautics  Document  Order-1 7 8B  and  related  documents  provide  guidelines  by  which 
to  qualify  these  tools.  However,  current  regulations,  business  models,  and  industry  practice  make  this  goal  difficult  to  accomplish. 

This  article  discusses  the  qualification  of  development  tools  and  the  potential  impact  of  this  process  on  the  aviation  industry. 


Software  development  tools  are  com¬ 
puter  programs  that  help  developers 
create  other  programs.  Such  tools  have 
been  in  use  since  the  early  days  of  com¬ 
puting  to  improve  the  efficiency  of  the 
development  process  by  automating  mun¬ 
dane  translation  operations  and  bringing 
the  level  of  abstraction  closer  to  the  appli¬ 
cation  engineer.  Nowadays,  development 
tools  are  used  in  a  variety  of  safety- critical 
applications,  including  the  aviation,  auto¬ 
motive,  space,  nuclear,  railroad,  medical, 
and  military  industries,  and  contribute  to 
the  risks  associated  with  using  respective 
products.  Despite  these  risks  to  society, 
development  tools  are  rarely  qualified  in  a 
sense  comparable  to  product  certification 
in  regulated  industries.  The  objective  of 
this  article  is  to  look  at  the  current  state  of 
the  tool  qualification  process,  identify  the 
issues,  and  propose  recommendations  for 
potential  improvement,  focusing  on  the 
aviation  industry. 

System  Certification  Versus 
Software  Tool  Qualification 

Certification  of  airborne  equipment  is  typ¬ 
ically  achieved  through  the  Federal 
Aviation  Administration  (FAA)  authoriza¬ 
tion  of  a  type  certificate  (the  entire  air¬ 
craft),  supplemental  type  certificate  (new 
equipment  in  a  specific  aircraft),  or  a  tech¬ 
nical  standard  order  (minimum  perfor¬ 
mance  standard  for  materials,  parts,  and 
appliances  used  on  civil  aircraft).  A  special 
committee  (SC- 145)  of  the  Radio  Tech¬ 
nical  Commission  for  Aeronautics  (RTCA) 
convened  in  1980  to  establish  guidelines 
for  developing  airborne  systems.  The 
report  “Software  Considerations  in  Air¬ 
borne  Systems  and  Equipment  Certifica¬ 
tion”  was  published  in  January  1982  as  the 
RTCA  Document  Order  (DO)-178  (and 
revised  as  DO-178A  in  1985). 

Due  to  rapid  advances  in  technology, 
the  RTCA  established  a  new  committee 


(SC-167)  in  1989  with  the  objective  of 
updating  the  DO-178A  by  focusing  on 
five  areas:  documentation  integration  and 
production,  system  issues,  software  devel¬ 
opment,  software  verification,  and  soft¬ 
ware  configuration  management  and  soft¬ 
ware  quality  assurance.  The  resulting  doc¬ 
ument,  DO-178B,  provides  guidelines  for 
applicants  developing  software-intensive 
airborne  systems  [1,2].  It  discusses  objec¬ 
tives  that  need  to  be  met  to  show  that  the 
software  development  process  provides 
specified  levels  of  safety  assurance.  It  also 
describes  the  processes  and  means  of 
compliance. 

Systems  are  categorized  by  DO-178B 
as  meeting  safety  assurance  levels  A 
through  E  based  on  their  criticality  in  sup¬ 
porting  safe  aircraft  flight.  The  level  A  sys¬ 
tem  is  the  most  critical:  The  failure  of 
such  a  system  could  result  in  a  catastroph¬ 
ic  failure  condition  for  the  aircraft.  The 
level  E  system  is  the  least  critical:  Such  a 
system  has  no  effect  on  the  operational 
capability  of  the  aircraft  or  pilot  workload. 
Although  the  RTCA  DO-178B  is  the  lead¬ 
ing  source  of  guidelines  for  software 
developers  engaged  in  such  system  con¬ 
struction,  two  other  documents  have  criti¬ 
cal  bearing  on  the  subject.  RTCA  DO- 
248B  [3]  clarifies  some  of  the  misinterpre¬ 
tation  of  the  DO-178B.  The  FAA  Order 
8110.49  compiles  a  variety  of  guidelines 
related  to  the  use  of  software  in  airborne 
systems.  Chapter  9  is  specifically  dedicated 
to  tool  qualification  [4]. 

A  key  component  of  the  updated  ver¬ 
sion  of  DO-178B  is  the  concept  of  tool 
qualification  elaborated  in  Section  12. 
Qualification  is  a  supplementary  process 
that  the  applicant  may  elect  to  follow  in 
the  course  of  certifying  an  airborne  sys¬ 
tem.  According  to  the  definition  given  in 
DO-178B,  tool  qualification  is  defined  as, 
“The  process  necessary  to  obtain  certifica¬ 
tion  credit  for  a  software  tool  within  the 


context  of  a  specific  airborne  system.”  It  is 
the  certification  authority  that  decides  on 
the  outcome  of  the  qualification  process. 
Moreover,  qualification,  if  claimed,  is  a 
requirement  in  getting  a  system  certified. 

Types  of  Software 
Development  Tools 

DO-178B  differentiates  between  verifica¬ 
tion  tools  that  cannot  introduce  errors  but  may 
fail  to  detect  them  and  development  tools 
whose  output  is  part  of  airborne  software  and 
thus  can  introduce  errors.  There  is  a  signifi¬ 
cant  amount  of  effort  involved  to  qualify 
a  verification  tool,  and  much  more  to 
qualify  a  development  tool.  However, 
numerous  development  tools  have  been 
used  successfully  in  many  certified  pro¬ 
jects  without  being  qualified.  To  define  a 
subject  matter  more  narrowly,  we  need  to 
take  a  closer  look  at  the  entire  domain  of 
software  development  tools. 

The  landscape  of  modern  software 
development  tools  is  very  broad,  as  illus¬ 
trated  in  Figure  1  (see  page  20).  Following 
the  traditional  model  of  the  development 
process  from  requirements  to  implemen¬ 
tation,  we  can  identify  the  following: 

•  The  requirements  category  that  includes 
tools  used  early  in  the  life  cycle  to 
identify  and  specify  the  software 
requirements. 

•  The  design  category  that  includes  tools 
allowing  developers  to  create  architec¬ 
tural  and  detailed  design  of  the  soft¬ 
ware  in  a  notation  of  their  choice  sup¬ 
ported  by  the  tool;  often  in  this  cate¬ 
gory,  tools  translate  the  model  to 
source  code. 

•  The  implementation  category  that 
includes  all  support  required  to  trans¬ 
late  the  computer  code  and  transfer  it 
to  the  target  computer. 

As  illustrated  in  Figure  1,  three  other 
categories  of  tools  can  be  identified:  those 
related  to  analysis ,  testing,  and  target. 
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Figure  1 :  Software  Tool  Categories 

However,  for  this  article,  we  will  focus  pri¬ 
marily  on  the  tools  used  in  the  design 
phase,  the  central  component  of  the  soft¬ 
ware  development  life  cycle.  They  reflect 
two  diverse  viewpoints  on  real-time,  safety- 
critical  systems  development,  which  result 
from  different  developers’  backgrounds: 

•  Control  engineers  consider  a  system  to 
be  a  dynamic  model  consisting  of  well- 
defined  blocks  of  specific  functionality 
(logic,  arithmetic,  dynamic).  The  func¬ 
tional  paradigm  of  the  model  is  the 
basis  for  system  simulation  and  analysis 
of  its  behavior.  Subsequently,  the  model 
can  be  translated  automatically  into  an 
equivalent  code,  typically  without  any 
additional  developer’s  involvement. 

•  Software  engineers,  on  the  other  hand, 
are  familiar  with  the  concepts  of  oper¬ 
ating  systems,  programming  languages, 
software  development  methodologies, 
and  notations.  The  graphic  notations 
(classes,  packages,  states,  transitions, 
events)  allow  developers  to  represent 
the  structure  and  behavior  of  the  target 
system  software  as  a  set  of  components 
that  can  be  translated  into  programming 
constructs  (data  structures,  objects, 
functions,  etc.)  using  the  automatic  code 
generation  functionality  of  the  tool. 
Consequently,  the  software  design 

tools,  which  assist  developers  in  translat¬ 
ing  the  software  requirements  into  source 
code,  can  be  categorized  into  two  groups: 
(a)  a  function-based ,  block-oriented  approach 
applied  by  control  and  system  engineers, 
and  (b)  a  structure-based ,  object-oriented 
approach  applied  by  computer  scientists 
and  software  engineers. 

The  Qualification  Process 

A  typical  use  for  a  design  tool  (software 


producer)  is  to  transform  an  input  artifact 
into  output,  thus  creating  another  software 
artifact.  The  current  process  mandates  ver¬ 
ification  after  each  transformation.  If  this 
transformation  has  an  impact  on  the  final 
airborne  product,  the  producer  needs  to  be 
qualified,  but  only  if  the  transformation 
output  would  not  be  verified  and  the  trans¬ 
formation  leads  to  elimination,  reduction, 
or  automation  of  any  of  the  DO-178B 
processes.  The  conditions  under  which  a 
development  tool  requires  qualification  are 
presented  in  Figure  2  [4] . 

Software  development  tool  qualifica¬ 
tion  is  attempted  only  as  an  integral  com¬ 
ponent  of  a  specific  application  program 
requiring  the  FAA’s  certification.  The  soft¬ 
ware  tools  to  be  used  are  referenced  with¬ 
in  the  Plan  for  Software  Aspects  of 
Certification  (PSAC)  and  the  Software 
Accomplishment  Summary  documents  of 
the  original  certification  project.  If  devel¬ 
opment  tool  qualification  is  required,  the 
applicant  should  present  for  review  the 
Tool  Operational  Requirements  (TOR)  -  a 
document  describing  tool  functionality, 
environment,  installation,  operation  man¬ 
ual,  development  process,  and  expected 
responses  (also  in  abnormal  conditions). 

Two  documents  must  be  submitted  and 
approved:  a  Tool  Qualification  Plan,  and  a 
Tool  Accomplishment  Summary  as  de¬ 
scribed  in  [4].  To  make  an  argument  for 
qualification,  the  applicant  must  demon¬ 
strate  correctness,  consistency,  and  com¬ 
pleteness  of  the  TOR  and  show  that  the 
tool  complies  with  its  TOR.  This  demon¬ 
stration  may  involve  a  trial  period  during 
which  a  verification  of  the  tool  output  is 
performed  and  tool-related  problems  are 
analyzed,  recorded,  and  corrected. 

Other  data  required  for  review  include 


a  Tool  Configuration  Management  Index, 
Tool  Development  Data,  Tool  Verification 
Records,  Tool  Quality  Assurance  Records, 
Tool  Configuration  Management  Records, 
etc.  These  requirements  are  also  described 
in  [4] .  Tool  qualification  data  are  approved 
only  in  the  context  of  the  overall  software 
development  for  the  specific  system 
where  the  intention  to  use  the  tool  is  stat¬ 
ed  in  the  PSAC.  The  tool  itself  does  not 
receive  a  separate  qualification  stamp  of 
approval.  Therefore,  using  the  tool  on 
another  system/project  requires  a  separate 
qualification,  although  some  qualification 
credits  may  be  reused. 

Surveys  requesting  which  tools  are  used 
by  industry  were  conducted  at  two  national 
conferences:  the  2002  FAA  National 
Software  Conference  and  the  2004  Embry 
Riddle  Aeronautical  University/ FAA  Soft¬ 
ware  Tool  Forum.  In  addition,  two  follow¬ 
up  e-mail  solicitations  were  sent  to  more 
than  500  professionals  working  on  air¬ 
borne  systems.  These  surveys  and  solicita¬ 
tions  resulted  in  a  relatively  small  sample  of 
responses  that  did  not  provide  a  base  for 
statistically  significant  results.  The  com¬ 
ments  included  industry  discouragement 
regarding  the  rigor  of  development  tool 
qualification,  and  a  justified  perception  of 
the  extensive  cost  of  qualification. 

Potential  solutions  to  assist  in  commer¬ 
cial  off-the-shelf  (COTS)  development 
tool  qualification  included  extensive  ven¬ 
dor  collaboration  and  using  alternate  means 
allowed  in  DO-178B.  The  limited  feedback 
shows  that  there  has  been  interest  in  qual¬ 
ifying  software  development  tools  classi¬ 
fied  in  the  function-based/block-oriented 
category,  which  cannot  be  said  about 
structure-based/ object-oriented  tools. 

A  short  list  of  qualified  development 
tools  includes  code  generators  (Gener¬ 
ation  Automatique  de  Logiciel  Avionique, 
Graphical  Processing  Utility,  Virtual 
Application  Prototyping  System  Code 
Generator,  Safety  Critical  Avionic 
Development  Environment  Qualifiable 
Code  Generator)  and  configuration¬ 
scheduling  table  generators  (Universal 
Table  Builder  Tool,  Configuration  Table 
Generation  Tool),  most  of  them  being  in- 
house  products.  According  to  several 
informal  exchanges  with  industry,  many  of 
the  modern  COTS  software  development 
suites  actually  have  been  used  in  the  cre¬ 
ation  of  software  artifacts  on  certified 
projects  without  going  through  the  quali¬ 
fication  process. 

Problems  With  Development 
Tool  Qualification 

It  is  clear  that  qualification  of  develop- 
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ment  tools  is  an  option  rarely  exercised  in 
the  airborne  software  industry  In  fact, 
one  could  argue  that  qualification  of 
development  tools  is  not  a  viable  option. 
Current  interpretation  of  applicable 
guidelines  makes  development  tool  quali¬ 
fication  a  proposition  that  is  not  practical 
from  a  managerial  viewpoint ,  and  not  easy 
from  a  technical  viewpoint. 

Managerial  Viewpoint 

The  first  group  of  problems  is  of  a  regu¬ 
latory  and  managerial  nature.  The  major 
hurdle  is  the  current  state  of  regulations 
and  guidelines.  The  secondary  obstacle  is 
the  business  model  and  lack  of  incentives, 
in  particular  the  prohibitive  cost  of  tool 
qualification.  The  existing  tools,  often 
used  in  certification  projects,  do  not  have 
appropriate  data  to  support  arguments 
about  meeting  the  objectives  of  DO- 
178B.  The  applicant  team’s  intent  is  to  cer¬ 
tify  the  product  rather  than  expand  effort 
and  qualify  the  tool.  The  tool  vendor  does 
not  see  the  business  advantage  of  qualify¬ 
ing  a  tool  while  disclosing  proprietary 
information  to  potential  competitors. 

Development  tool  qualification 
requires  close  collaboration  between  the 
tool  vendor  and  the  applicant.  This  is  the 
reason  why  in-house  tools  are  more  likely 
to  be  qualified.  Internal  trade  studies  [5] 
have  shown  that  the  cost  of  development 
tool  qualification  is  significantly  higher 
than  the  cost  of  verification  tool  qualifica¬ 
tion.  The  use  of  qualified  verification 
tools  can  result  in  fast  savings  on  the  first 
program  where  they  are  introduced.  In 
contrast,  the  use  of  qualified  development 
tools  may  require  several  programs  to 
make  up  the  cost. 

The  intellectual  property  rights  may 
need  to  be  waived  by  the  vendor  to  achieve 
qualification.  The  tool  cannot  be  qualified 
as  standalone,  but  only  within  the  scope  of 
a  particular  certification  project.  The  tools 
that  could  be  considered  for  qualification 
are  very  simple:  typically  in-house  created 
utilities  where  the  applicant  holds  all  intel¬ 
lectual  property  rights,  maintains  all  tool 
development  data,  and  can  reuse  the  tool 
software  artifacts  on  consecutive  projects. 
The  qualification  is  accomplished  within 
the  specific  certification  project  and  thus  is 
not  clearly  visible  from  the  outside  as  devel¬ 
opment  tool  qualification. 

Technical  Viewpoint 

The  second  group  of  problems  is  related 
to  technical  aspects.  According  to  the  DO- 
17813  interpretation,  the  development  tool 
needs  to  be  qualified  to  the  same  level  of 
scrutiny  as  the  appropriate  application  it  is 
helping  to  develop.  However,  there  is  a  sig¬ 


nificant  difference  between  tool  software 
and  application  software.  Applications  run 
on  a  target  computer  while  tools  operate 
on  a  general-purpose  workstation,  typical¬ 
ly  closely  interacting  with  a  COTS  operat¬ 
ing  system  and  conventional  programming 
environment.  Considering  this,  several 
DO-178B  objectives  are  not  applicable  to 
tool  software  and  thus  cannot  be  met. 
There  is  also  no  general  agreement  on 
what  metrics  would  allow  developers  to 
carry  an  independent  tool  assessment  [6]. 

One  often-repeated  statement  regard¬ 
ing  development  tool  qualification  is  the 
requirement  that  “only  deterministic  tools 
can  be  qualified.”  The  DO-178B  refers  to 
determinism  as  “...  tools  which  produce 
the  same  output  for  the  same  input  data 
when  operating  in  the  same  environment.” 
The  definition  does  not  take  into  account 
how  the  output  is  generated.  By  this  defin¬ 
ition,  one  may  interpret  that  it  is  not 
required  to  provide  proof  on  the  internal 
behavior  of  a  tool.  An  example  of  this  can 
be  memory  use  for  a  tool  running  on  the 
host  workstation  in  a  multitasking,  multi¬ 
user,  networked  environment.  The  prob¬ 
lem  is  to  define  what  the  object  code  for  a 
tool  is.  Does  it  include  the  operating  system 
(OS)  of  the  host  workstation?  A  tool  clear¬ 
ly  needs  to  make  explicit  calls  to  the  OS 
routines,  and  any  verification  of  these 
would  require  full  visibility  of  the  host’s  OS 
and  related  high  assurance  of  its  operation. 

The  main  function  of  a  software 
development  tool  is  to  transform,  i.e., 
translate  an  input  artifact  into  output.  This 
is  why  the  qualification,  if  applicable, 
should  be  focused  on  this  translation 
component  of  the  tool  functionality. 
However,  modern,  complex  software 
development  tools  provide  a  variety  of 
other  functions  that  are  not  directly  relat¬ 


ed  to  the  translation  process.  The  transla¬ 
tion  component  is  hidden  deep  inside  the 
tool,  which  causes  problems  with  tool 
qualification.  Typically,  there  is  no  access 
to  a  COTS  tool’s  life-cycle  data,  which 
describe  the  tool’s  requirements,  design, 
and  code.  Unless  the  tool  has  been  devel¬ 
oped  in-house,  the  qualification  efforts 
may  be  doomed. 

Potential  Solutions 

The  qualification  of  a  stand-alone  devel¬ 
opment  tool  is  not  feasible  in  the  strict 
sense  of  existing  guidelines.  Such  concepts 
as  component-based  software,  software 
reuse,  and  service  history  should  be 
explored  [7]  to  identify  the  feasibility  of 
such  qualification.  The  issues  of  tool  ver¬ 
sion  control  and  the  precise  definition  of 
operational  environment,  constraints,  and 
limitations  are  the  basis  for  starting  discus¬ 
sion  about  solutions  to  tool  qualification. 
The  availability  of  extensive  tool  software 
development  data,  often  scarce  for  COTS 
products,  may  be  a  challenge  to  ever 
accomplish  COTS  tool  qualification  [8]. 

It  could  be  conceivable  to  create  an 
independent  lab  dedicated  to  tool  qualifi¬ 
cation  and  encourage  commercial  vendors 
to  submit  their  product  for  assessment.  A 
similar  approach  is  known  from  other 
areas  of  verification  and  validation  [9,  10]. 
Another  idea  would  be  to  require  certified 
product  applicants  to  disclose  information 
regarding  the  development  tool  use  and 
qualification  effort  by  creating  an  FAA- 
sponsored  database  for  DO-178B  certi¬ 
fied  products.  This  could  face  serious 
objections  from  industry  due  to  an  appre¬ 
hensiveness  to  disclose  any  information, 
which  may  result  in  the  loss  of  commer¬ 
cial  advantage.  It  would  be  possible  to 
research  a  potential  for  development  tool 


Figure  2:  Conditions  When  a  Software  Development  Tool  Requires  Qualification 
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qualification  using  an  approach  different 
than  the  one  outlined  in  Section  12.2  of 
DO-178B.  Service  history  and  formal 
methods  could  both  be  potential  options. 

It  appears  that  the  industry  has  a 
pressing  need  to  come  up  with  methods  to 
audit  a  tool  that  is  independent  of  the  spe¬ 
cific  program  and  applications  using  it. 
This  would  require  updating  the  guidelines 
to  consider  a  model-driven  development 
paradigm,  redefine  the  qualification 
process,  and  allow  flexibility  regarding 
qualification  to  be  less  dependent  on  the 
application  program  using  the  tool.  A 
more  streamlined  method  to  qualify  devel¬ 
opment  tools  and  to  keep  them  current  as 
technology  advances  would  be  useful. 
Better  guidance  on  how  to  apply  service 
history  and  how  to  address  what  has  to  be 
done  for  incremental  tool  changes  would 
also  be  needed.  These  and  other  issues 
have  been  discussed  at  the  recent  Tools 
Forum  [11].  The  RTCA  convened  another 
special  committee  (SC-205)  with  a  charge 
to  recommend  modifications  to  the  exist¬ 
ing  DO-178B.  The  qualification  of  soft¬ 
ware  tools  is  being  discussed  and  some 
changes  may  be  forthcoming.  ♦ 
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Using  Line  of  Balance  to  Track  the  Progress  of 
Fixing  Trouble  Reports 


Eduardo  Miranda 

Independent  Consultant 

You  are  the  project  manager  of  a  large  project  and  testing  is  uncovering  faults,  trouble  reports  are  starting  to  pile  up  and  the 
release  date  is  coming  soon.  Are  they  going  to  be  fixed  on  time ?  What  could  you  do  to  help?  Are  there  any  bottlenecks ?  Where 
should  you  assign  more  resources ?  Does  this  scenario  sound familiar ?  Dave  you  been  there ?  This  article  will  explain  how  you 
can  answer  these  questions  by  using  an  old  method  called  Tine  of  Balance  in  a  new  way. 


Line  of  balance  (LOB)  was  devised  by 
the  members  of  a  group  headed  by 
George  E.  Fouch  during  the  1940’s  to 
monitor  production  at  the  Goodyear  Tire 
&  Rubber  Company  [1].  It  was  also  suc¬ 
cessfully  applied  to  the  production  plan¬ 
ning  and  scheduling  of  the  huge  Navy 
mobilization  program  of  World  War  II 
and  during  the  Korean  hostilities.  Today, 
LOB  application  has  been  further  expand¬ 
ed,  making  it  suitable  for  a  whole  spec¬ 
trum  of  activities  ranging  from  research 
and  development  through  job -shop  and 
process  flow  operations. 

In  the  context  of  managing  a  software 
project,  the  LOB  technique  offers  two 
main  advantages  over  the  traditional  Open 
Trouble  Reports  (TRs)  Chart  [2] : 

•  It  allows  project  managers  to  see,  in 
the  middle  of  a  project,  whether  they 
can  meet  the  schedule  if  they  continue 
working  as  they  have  been. 

•  It  exposes  process  bottlenecks,  allow¬ 
ing  the  project  manager  to  focus  on 
those  points  responsible  for  slippage. 

The  Open  TRs  Chart 

To  answer  some  of  the  questions  raised  at 
the  beginning  of  this  article,  project  man¬ 
agers  usually  resort  to  the  Open  TRs  Chart 
shown  in  Figure  1  or  a  variation  of  it. 

The  Open  TRs  Chart  shows  the  cumu¬ 
lative  number  of  TRs  written  over  time, 
and  its  breakdown  into  open  and  closed 
TRs.  As  the  project  progresses,  the  closed 
line  should  converge  toward  the  total  line 
and  the  open  line  towards  zero.  A  closed 
line  that  is  not  converging  fast  enough 
toward  the  total  or  an  open  line  that  does 
not  approach  zero  signals  to  the  project 
manager  the  need  to  devote  additional 
resources  to  fix  problems. 

Variations  of  the  chart  include  show¬ 
ing  a  more  detailed  breakdown  of  the  TR 
status,  and  ratios  between  total  and  open 
TRs  [2,  3]. 

Despite  all  its  usefulness,  the  Open  TR 
Chart  lacks  predictive  ability  and  fails  to 
take  advantage  of  past  and  present  perfor¬ 
mance  data  and  TRs  closure  targets;  i.e., 
how  many  TRs  should  be  in  a  given  state 


by  a  given  time  to  meet  the  project  dead¬ 
lines.  In  other  words,  although  the  chart 
will  give  the  project  manager  a  gut  feeling 
about  the  situation,  it  would  not  answer 
the  questions  of  where  are  we  in  relation 
to  where  we  are  suppose  to  be,  or  how 
much  better  we  should  be  doing  to  get 
where  we  want  to  get  by  the  time  we  want. 

TheTR  Life  Cycle 

Typically,  a  TR  will  go  through  a  number 
of  stages  or  states  since  it  is  reported  until 
it  is  closed  (see  Figure  2,  page  24).  Each  of 
these  states  corresponds  to  a  milestone  in 
the  process  of  answering  a  TR  into  which 
the  organization  or  project  manager  wants 
to  have  visibility  to  evaluate  progress,  i.e., 
how  many  TRs  have  been  reported,  how 
many  of  the  reported  TRs  have  been  ana¬ 
lyzed,  how  many  of  the  analyzed  were 
rejected  and  so  on.  Elemental  states  could 
be  grouped  into  super  sets  for  reporting 
purposes,  i.e.,  while  the  project  manager 
might  be  interested  in  how  many  have 
been  analyzed,  assigned,  implemented,  or 
integrated  the  steering  group  overseeing 
the  project  might  only  been  interested  in 
how  many  TRs  were  reported,  how  many 
were  closed,  and  how  many  were  still 
pending. 

Most  defect  tracking  systems  will 
implement  this  model  or  some  variation 


of  it,  time  stamping  each  TR  as  they  tran¬ 
sition  between  states.  This  last  feature 
would  allow  the  organization  to  produce 
the  lead-time  information  required  by  the 
LOB  method. 

In  addition  to  the  state  and  timing 
information,  the  TR  includes  other  data 
such  as  the  severity  of  the  problem.  This 
information  could  be  used  to  filter  the  TR 
data  and  apply  the  LOB  method  to  a  sub¬ 
set  of  all  the  TRs  reported  and  in  the  pri¬ 
oritization  of  which  TRs  to  fix  first. 

The  LOB  Method 
Applied  to  TRs 

The  LOB  method  consists  of  the  follow¬ 
ing  elements  [4] : 

•  A  number  of  control  points  and  their 
lead  times  to  closing  as  illustrated  in 
Figure  3  (see  page  24),  at  which 
progress  is  to  be  monitored. 

•  An  Objective  Chart  or  target  plan  dis¬ 
playing  the  cumulative  closing  sched¬ 
ule  as  planned  by  the  project  manager 
to  meet  a  set  deadline  (Figure  4). 

•  The  TRs  Status  Chart  (see  Figure  5, 
page  25),  which  shows  the  actual  num¬ 
ber  of  TRs  that  have  passed  through  a 
given  control  point  versus  the  number 
that  should  have  been  passed  (the 
LOB)  according  to  the  plan. 

The  information  contained  in  the 
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Figure  2:  Typical  TR  Life  Cycle 


Figure  3:  The  Process  of  Solving  a  TR  and  Its 
Corresponding  Lead  Times 


Control  Point 

Time  in  State 

Lead  Time 

Reported 

5 

5  +  25  =  30 

Analyzed 

6 

6  +  19  =  25 

Assigned 

8 

8  +  11  =  19 

Implemented 

6 

6  +  5  =  11 

Integrated 

5 

5  +  0  =  5 

Verified 

0 

0 

Table  1 :  Lead-Time  Calculations 


Objective  Chart,  together  with  the  lead- 
time  information  is  used  to  calculate  how 
many  TRs  should  be  in  a  given  state  at  a 
given  time. 

Control  Points 

In  LOB  terminology,  a  control  point  is  a 
milestone  or  event  that  the  project  manager 
wants  to  monitor.  In  the  context  of  track¬ 
ing  TRs,  the  control  points  and  states  in  the 
TR  life  cycle  would  most  likely  coincide,  but 
this  is  not  necessary1.  For  example,  the  pro¬ 
ject  manager  might  not  find  it  useful  to 
track  TRs  in  the  rejected  state  and  so  this 
state  would  not  be  considered  a  control 
point.  The  lead  time  for  a  control  point  is 
calculated  using  the  following  formula: 

LeadTimen=0 

LeadTlme^^  ^.^TlmelnStateq+LeacfTlmeq+i 

Assuming  that  the  median2  times  a  TR 
spends  in  a  given  state  are  those  shown  in 
Table  1,  the  lead-time  calculations  will 
yield  the  results  illustrated  in  Figure  3. 

The  Objective  Chart 

The  Objective  Chart  shows  cumulative,  to  he 
verified  TRs  on  the  vertical  scale  and  dates  of 
achievement  along  the  horizontal  scale.  The 
chart  might  also  include  a  display  of  the 
achievements  so  far. 

The  Objective  Chart  in  Figure  4  shows 
that  the  project  manager  has  committed  to 
close  50  TRs  by  the  end  of  September,  80 
by  the  end  of  November,  and  150  by  the 
beginning  of  the  following  year.  The  chart 
also  shows  that  as  of  mid-December 
progress  is  slightly  behind  with  the  project 
delivering  around  75  fixed  TRs  instead  of 
the  80  promised. 

TR  Status  Chart 

The  TR  Status  Chart  provides  quantitative 


information  with  regards  to  progress,  and 
whether  or  not  there  is  a  bottleneck  on  the 
process. 

The  chart  portrays  the  actual  number 
of  TRs  that  have  passed  through  each 
control  point  against  the  number  that 
should  have  been  passed  according  to  the 
plan.  These  last  quantities  are  called  the 
LOB.  The  difference  between  the  LOB 
and  the  top  of  the  bar  for  each  control 
point  is  the  number  of  TRs  behind  or 
ahead  of  schedule. 

Notice  that  the  shape  of  the  LOB  will 
change  daily  even  if  there  are  no  new  TRs 
reported,  since  its  calculation  depends  on 
the  planned  curve  of  the  Objective  Chart 
and  the  status  date. 

The  TR  Status  Chart  shows  that  there 
are  almost  180  TRs  reported  so  far,  30 
more  than  what  were  planned  to  fix 
according  to  the  Objective  Chart.  This  sig¬ 
nals  the  need  to  update  the  plan.  It  also 
tells  us  that  TR  implementation  is  on  track 
as  the  actual  column  and  the  LOB  line  for 
that  control  point  coincide,  but  that  we  are 
falling  behind  in  their  integration  and  ver¬ 
ification.  This  suggests  that  adding  more 
people  to  implementation  activities  will 
not  help  recoup  the  delay,  but  that  addi¬ 
tional  resources  could  be  used  in  integra¬ 
tion  and  verification  activities. 

The  LOB  for  each  control  point  is  cal¬ 
culated  as  follows: 

ti<  t  <t2 
t2<  t  <t3 


tm<  t  <tm+1 


ai  +  bit 

a2  +  b2t 


LOB, 


q=1,2,. 


3m  +  bmt 


bi=  YtlI 

t+i  -  ti 


To  Be  Verified 


Figure  4:  The  Plan  Proposed  by  the  PM  to  Clear  the  TR  Backlog 


yi  and  y+i  are  the  number  of  TRs  to  be 
fixed  by  time  t  and  tm  respectively,  as 
planned  by  the  project  manager  and 
captured  in  the  objective  chart. 

ai  =  yi  -  bits 

t  =TimeNow+LeadTimeq 

The  idea  behind  the  procedure  is  simple. 
If  it  takes  an  average  of  10  days  for  a  TR 
to  go  from  a  given  state  to  the  completion 
state,  today’s  status  for  that  state  should 
be  equal  to  the  number  of  TRs  that  would 
have  to  be  completed  according  to  the 
plan  1 0  days  from  now.  See  Figure  6  for  a 
graphical  example. 

In  Figure  6,  the  chart  on  the  left  shows 
the  planned  line  from  Figure  4,  while  the 
chart  on  the  right  shows  the  scheduled 
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Progress  at  Status  Points  as  of  12/12/2004 
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Figure  5:  Trouble  Reports  Status  Chart 


2.  The  median  is  preferred  to  the  arith¬ 
metic  mean  (average)  to  prevent  rare  but 


complex  TRs  from  skewing  the  value  of 
the  statistic  to  the  right. 


About  the  Author 


line  from  Figure  5.  We  obtained  Figure  5’s 
scheduled  line  by  finding  the  interception 
between  the  TimeNoiv  line  and  the  curve  in 
the  objective  chart  (the  ai  +b2t,...,am  +  bmt 
function  above),  which  yields  the  value  for 
the  Verified  Control  Point,  that  is  the 
number  of  TRs  that  should  be  on  that 
state  as  of  Dec.  12,  2004.  The  interception 
between  the  curve  and  the  line  at  TimeNoiv 
+  TeadTirneimpkmnted  yields  the  LOB  value  for 
the  Implemented  Control  Point. 

Summary 

By  providing  a  credible  early  warning 
about  bottlenecks  in  the  process  of  fixing 
TRs,  the  LOB  method  helps  project  man¬ 
agers  take  corrective  actions  such  as  allo¬ 
cating  more  resources  or  prioritizing  the 
work  when  there  is  still  time  to  do  it. 

In  terms  of  the  data  required  to  imple¬ 
ment  the  LOB  technique,  most  of  it 
should  be  readily  available  from  your 
defect  tracking  system  or  could  be  derived 
from  it  with  a  few  calculations  implement¬ 
ed  in  Excel  or  any  other  spreadsheet.  ♦ 
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Notes 

1.  The  control  points  are  likely  to  be  a 
subset  of  the  TR  states.  To  avoid  con¬ 
fusion,  do  not  create  additional  control 
points. 
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Letters  to  the  Editor 


Dear  CROSSTALK  Editor, 

In  CROSSTALK’S  February  issue,  Ken  Schwaber  commented 
on  my  December  article,  titled  “Agile  Software  Development 
for  the  Entire  Project.”  I  am  in  firm  agreement  with  him  theo¬ 
retically  and  philosophically  However,  the  open  water  reality  of 
working  in  agile  software  development  brings  home  how  diffi¬ 
cult  agile  software  can  sometimes  become.  In  both  thought  and 
practice,  it  requires  us  to  be  open  and  willing  to  change  to  keep 
up  with  evolution.  After  all,  if  there  is  one  thing  that  agile 
processes  tell  us,  it  is  to  embrace  change.  Perhaps  it  requires 
more  change  of  the  traditional  mindset  than  one  of  the  orga¬ 
nization. 

As  such,  the  agile  community  has  been  working  diligently  to 
best  understand  how  various  roles  —  such  as  test  —  fit  into  agile 
software  development  processes.  If  one  has  the  opportunity  to 
attend  an  agile  conference,  one  notices  entire  tracks  devoted  to 
testing  the  results  of  agile  processes.  Indeed,  most  organizations 
end  up  integrating  testing  on  their  own,  using  a  Chinese  menu 
approach  [Bob  Martin,  keynote  Agile  Conference  2005]  where  they 
mix  and  match  various  agile  practices  to  construct  their  own 
processes. 

Working  with  on-site  customers  is  always  our  best-case  sce¬ 
nario.  The  feedback  is  direct,  constant,  and  immediate. 
Realistically,  however,  many  projects  cannot  bring  customers 
on-site  for  perfectly  good  reasons.  After  all,  customers  have 
jobs  too.  What  do  we  say  to  those  projects?  You  can’t  be  agile? 
No,  we  shift  a  little  and  bring  in  roles  to  go  to  work  with  the 
customers  on  their  site.  Customer  validation  is  an  extremely 
important  part  of  any  real-world  agile  process. 

This  article  demonstrates  the  need  for  a  next  generation 
(XP  is  already  on  version  2.0)  of  agile  processes  with  ways  of 
dealing  with  customers  who  cannot  spend  time  on-site,  larger 
agile  projects,  distributed  projects,  and/or  the  need  for  testing 
of  the  delivered  systems  on  behalf  of  the  customer.  You  can 
consider  any  one  of  these  conditions  a  barrier  to  using  some  of 


the  first-generation  processes.  Of  course,  these  new  processes 
will  have  to  continue  to  promote  adaptive,  self-managing  teams 
in  a  team-oriented  environment. 

I  wish  to  thank  Mr.  Schwaber  for  reading  my  article.  I  always 
appreciate  his  from-the-heart  feedback.  My  goal  is  to  strive  to 
be  true  to  agile  methodology,  while  addressing  the  real-life 
ongoing,  ever- changing,  ever-evolving  needs  of  our  customers 
and  our  exciting  industry. 

Granville  “Randy”  Miller 

Microsoft 

<randymi@microsoft.com> 

Dear  CROSSTALK  Editor, 

I  very  much  liked  the  BackTalk  column  about  software 
usability  titled  “Push  for  Cheese:  A  Metaphor  for  Software 
Usability”  by  Nicole  Radziwell  and  Amy  Shelton  in  the 
December  2005  issue  of  CROSSTALK.  It  was  very  thought- 
provoking.  I  frequently  get  annoyed  at  software  in  which  it  is 
either  too  hard  or  too  easy  to  invoke  a  selection  or  option,  and 
it  results  in  something  I  did  not  intend.  Your  suggestions  for 
how  to  overcome  this  problem  were  right  on.  It  is  very  challeng¬ 
ing  to  have  a  user  community  that  (a)  is  very  diverse  in  its  opin¬ 
ions  on  what  makes  for  good  software  usability  and,  (b)  as  they 
get  more  familiar  with  the  software,  their  preferences  on  how  it 
should  work  change. 

Thank  you  for  drawing  visibility  to  this  seemingly  simple  but 
in  actuality  very  complex  issue.  Hopefully  it  will  prevent  prob¬ 
lems  such  as  operator-machine  issues  coming  up  right  before 
going  to  final  system  test  that  should  have  been  addressed  much 
earlier  in  the  development. 

A1  Kaniss 
Naval  Air  Systems  Command 
<  alan.  kanis  s  @navy.mil> 


Web  Sites 


Tantara  Inc. 

www.tantara.ab.ca/info.htm 

Tantara  offers  practical  advice  for  software  process  improvement 
and  software  quality  assurance/management,  including  the 
International  Organization  for  Standardization  (ISO)  9001  (ISO 
9000-3,  TickIT),  ISO  12207,  ISO  15504  (Software  Process 
Improvement  and  Capability  dEtermination  [SPICE]),  the 
Software  Engineering  Institutes  Capability  Maturity  Model® 
(CMM®)  for  Software,  CMM  Integration^,  and  more.  Tantaras 
Web  site  is  updated  quarterly  and  offers  articles  (online,  compar¬ 
isons,  frameworks);  a  bulletin  board  of  events,  news,  and  statistics; 
job  aids;  and  books  and  training  materials. 

Software  Process  Improvement 
in  Regions  of  Europe 

www.cse.dcu.ie/ spire 

The  Software  Process  Improvement  in  Regions  of  Europe 
(SPIRE)  was  a  project  funded  by  the  European  Commission 
involving  partners  in  Austria,  Ireland,  Italy,  Sweden,  and 
Northern  Ireland.  SPIRE’s  objective  is  to  help  small  software 


development  units  (SUDs)  -  employing  a  software  staff  no  larg¬ 
er  than  50  —  to  get  business  benefits  from  investment  in  the 
Software  Process  Improvement  Network  (SPIN)  and  to  share 
their  experiences  with  others.  The  project  helped  nearly  60 
SUDs  in  SPIRE  to  carry  out  short,  cost-effective  improvement 
projects.  Their  experience  has  been  captured  in  the  European 
Analysis  Report,  which  provides  an  analysis  of  the  impact  of 
SPIN  within  SUDs  throughout  Europe,  and  identifies  trends 
and  key  features  for  future  planning  or  implementation. 

TickIT 

www.tickit.org 

TickIT  guides  the  developer  to  achieve  high-quality  software 
within  the  framework  of  the  ISO  9001.  TickIT  applies  to  all 
types  of  information  systems  that  involve  software  development 
processes  in  the  product  life  cycle.  Typical  systems  suppliers 
include  system  houses,  software  houses,  and  in-house  develop¬ 
ers.  TickIT  disciplines  are  also  relevant  to  the  development  of 
embedded  software.  A  full  definition  of  the  scope  of  TickIT  is 
included  in  the  TickIT  Guide,  available  on  the  Web  site. 
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How  to  Relate  Quality  and  Reuse  in  Evolving  Systems 

Dr.  Ronald  J.  Leach 

Howard  University 

This  article  suggests  a  model  to  predict  the  quality  of  software  developed  over  time,  where  the  reusable  components  are  also 
evolving  over  time. 


At  NASA  Goddard  Space  Flight  Center, 
we  analyzed  several  safety-critical  soft¬ 
ware  systems  with  sizes  ranging  from 
approximately  50,000  to  more  than  500,000 
lines  of  code  (LOC).  These  systems’  archi¬ 
tecture  included  a  reusable  software  core 
linked  to  mission- specific  software  that 
resulted  in  complete,  unique  ground  support 
systems  for  spacecraft  control.  Each  space¬ 
craft  required  both  new,  mission- specific 
source  code  and  an  interface  to  the  reusable 
core.  Because  of  technology  changes,  the 
reusable  core  itself  evolved,  increasing  in 
size  more  than  five-fold.  The  software  sys¬ 
tems  have  been  somewhat  superseded  by 
commercial  off-the-shelf  products. 

The  evolving,  reusable  core  had  a  much 
lower  defect  ratio  (defects  per  thousand 
LOC  [KLOC])  for  the  reusable  core  than 
similar  systems  in  the  same  application 
domain.  The  defect  ratios  of  0.034  and 
0.075  for  the  latest  two  versions  of  the 
reusable  core  discussed  here  are  far  lower 
than  the  range  of  0.12  to  1.89  for  similar 
systems.  The  extremely  low  defect  ratio  of 
the  core  was  even  more  impressive  in  view 
of  Les  Hatton’s  statement  that  very  few 
systems,  even  safety-critical  ones,  have  ever 
stayed  below  one  defect  per  KLOC  [1]. 

The  Model 

What  caused  the  unusually  high  quality  of 
the  evolving,  reusable  software  core?  Did 
the  reused  part  of  the  core  have  a  low 
defect  ratio  simply  because  it  had  been 
operational  for  so  long  in  a  safety- critical 
domain?  If  so,  we  should  expect  defects 
per  KLOC  to  follow  an  exponential  distri¬ 
bution  of  the  form  a  constant  times  e'K*T. 

On  the  other  hand,  was  the  new  code 
good  because  it  underwent  stringent 
analysis  and  testing?  If  so,  we  should 
expect  a  linear  relationship,  reflecting  the 
increase  in  code  size. 

These  two  questions  led  to  a  simple 
model  for  the  number  of  defects  in  each 
release  as  the  sum  of  an  exponential  dis¬ 
tribution  representing  defects  caused  by 
reused  code  and  a  linear  expression  repre¬ 
senting  the  defects  in  the  new  source 
code.  The  model  is: 

DEFECTS  =  (REUSED  *  e K  T  +  NEW)  *  M  * 
KLOC 

where. 


T  is  the  time  between  releases.  REUSED 
and  NEW  are  the  percentage  of  reused 
and  new  lines  of  code  in  a  release, 
respectively. 

KLOC  is  the  size  of  the  release. 

The  constants  K  and  M  are  discussed  later. 

Calibration  of  the  Model 

Calibration  of  the  model  requires  calcula¬ 
tion  of  the  constants  REUSED ,  NEW, ,  K , 
and  M.  We  already  know  KLOC  and  want 
to  estimate  DEFECTS.  The  overall  reuse 
percentages  between  releases  clustered 
around  80;  hence,  the  value  of  NEW 7 was 
.20  and,  thus,  our  the  model  became: 

DEFECTS  =  (.80  *  e K  T  +  .20)  *  M  *  KLOC 

To  compute  the  constant  K ,  we  entered  the 
defects  and  the  KLOC  for  each  release  into 
an  Excel  spreadsheet,  one  entry  per  release, 
and  then  computed  the  exponent  K  in  the 
exponential  distribution,  using  the  Excel 
LOGEST  function  to  compute  K.  This 
constant  was,  to  two  decimal  places,  0.79. 

The  constant  M  represents  the  change 
scale  between  LOC  and  DEFECTS.  To 
compute  M,  we  created  a  third  column  in 
the  Excel  spreadsheet,  representing  the 
difference  between  the  observed  number 
of  defects  for  previous  releases  and  what 
is  predicted  by  the  partial  formula  (.80  *  e 
•79*T  +  .20)  *  KLOC,  one  entry  per  release, 
and  then  computed  M  using  the  Excel 
SLOPE  function.  This  constant  M  was,  to 
two  decimal  places,  0.65,  making  the 
model  for  this  domain: 

DEFECTS  = 

(.80  *  e 79  T  +  .20)  *  0.65  *  KLOC 
Analysis  and  Future  Work 

It  is  possible  to  fine-tune  the  model  more 
than  what  we  presented  here.  Specific  val¬ 
ues  can  be  used  instead  of  the  averages  to 
improve  accuracy.  Using  different  multi¬ 
pliers  of  REUSED  and  NEW  can 
improve  estimation,  also. 

Measures  based  on  exponential  distrib¬ 
utions  frequently  are  used  to  assess  system 
quality,  reliability,  and  to  stop  testing  when 
the  expected  number  of  errors  remaining 
meets  the  objective  error  rate.  As  far  as  we 


know,  their  use  in  conjunction  with  soft¬ 
ware  reuse  to  predict  faults  is  new. 

Note  that  applying  reuse  measure¬ 
ments  to  evolving  code  is  somewhat  con¬ 
troversial.  Indeed,  Jeffrey  Poulin  [2]  and 
others  recommend  against  attempting 
software  reuse  when  the  underlying  code  is 
not  stable.  The  author  recommends  it  in 
certain  circumstances  [3],  while  others  are 
neutral  or  do  not  comment  on  the  issue. 
Readers  are  encouraged  to  provide  their 
experiences  via  e-mail  to  the  author. ♦ 
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When  Did  Six  Sigma  Stop  Being  a  Statistical  Measure? 

Joe  Schofield 

Sandia  National  Laboratories 

The  term  Six  Sigma  is  widely  used  as  an  approach  for  process  improvement  and  learning.  It  is  a  disciplined \  structured \  data- 
driven  methodology  to  solving  problems.  Along  the  path  to  popularity ,  Six  Sigma  lost  its  meaning  as  a  statistical  measure  and 
instead  inherited  the  meaning  of  merely  another  measurement  program.  Organisations  that  intend  to  employ  Six  Sigma  ought 
to  consider  which  definition  of  six  sigma  is  their  target:  a  process  improvement  approach,  or  a  statistical  measure  for  varia¬ 
tion.  This  article  explores  the  significance  of  the  differences  between  six  sigma  and  Six  Sigma.  Lead  on  if  you  dare. 


When  did  free  come  to  mean  free  after 
rebate ?  When  did  cost  become  the 
China  cost ?  And  when  did  Six  Sigma 
become  something  other  than  a  statistical 
notion?  Or  have  you  not  noticed  that  the 
term  Six  Sigma  no  longer  means  a  statistical 
measure  for  variation ?  For  every  organization 
that  attempts  to  use  six  sigma  as  a  statisti¬ 
cal  measure  of  process  improvement, 
three  other  organizations  use  it  merely  to 
describe  a  process  improvement  effort. 
Most  of  these  organizations  have  no 
intention  of  using  six  sigma  statistically, 
but  it  likely  impresses  the  folks  higher  up 
in  the  food  chain. 

Affixing  lean  to  the  term,  as  in  Lean 
Six  Sigma  (LSS),  is  currently  an  institu¬ 
tional  silver  bullet.  Do  not  feel  left  out  if 
you  have  not  been  exposed  to  LSS;  one 
Internet  search  found  only  about  125,000 
LSS-related  sites,  but  more  than  1.7  mil¬ 
lion  sites  for  Six  Sigma.  Very  few  of  these 
sites  advocate  six  sigma’s  statistical  mean¬ 
ing,  contributing  to  the  miscommunica- 
tion  regarding  Six  Sigma  processes. 

As  with  most  trendy  initiatives,  LSS 
has  it  own  status  symbols:  green  belts, 
black  belts,  and  an  assortment  of  colors 
and  variations  depending  on  the  accredit¬ 
ing  organization.  In  addition,  LSS  has  a 
lexicon,  words  like  kaizen,  kaikaku,  kan¬ 
ban  (yes,  there  are  more  than  just  k 
words).  There  is  one  more  Japanese  word 
that  the  LSS  industry  may  have  forgotten: 
it  is  muda,  or  the  word  for  waste.  Without 
applying  statistical  measurement,  organi¬ 
zations  may  be  wasting  their  process 
improvement  resources. 

The  application  of  LSS  may  bring 
numerous  well-intended  results,  including 
defect  reduction,  work  in  progress  reduc¬ 
tion,  cycle  time  reduction,  cost  savings, 
fewer  hand-offs  and  queues,  minimized 
changeover  time,  workload  leveling,  and 
more.  Organizations  in  pursuit  of  process 
improvement  are  often  well-advised  to 
consider  LSS  to  diagnose,  improve,  and 
measure  their  processes. 

Motorola  Corporation  gets  much  of 


the  credit  for  popularizing  Six  Sigma  and 
the  phrase  3.4  defects  per  million  —  the  battle 
cry  of  the  Six  Sigma  world.  Simply  re- stat¬ 
ed,  Six  Sigma  has  come  to  be  synonymous 
with  no  more  than  3.4  defects  per  million 
opportunities  (DPMO).  An  opportunity 
might  be  defined  as  a  keystroke  or  a 
mouse  click,  depending  on  whether  the 
process  being  measured  is  developing 
software  or  writing  an  article. 

Often,  the  value  3.4  DPMO  is  fol¬ 
lowed  with  a  footnote  or  an  asterisk;  the 
fine  print  typically  ignored.  Six  Sigma  pro¬ 
ponents  claim  that  the  3.4  DPMO  is  the 
long-term  process  performance  after  the 
occurrence  of  a  sigma  shift.  The  sigma  shift 
is  a  1.5  sigma  difference  from  6  to  4.5 
sigma  performance.  The  underlying 
assumption  is  that  short-term  perfor¬ 
mance  (of  say  6  sigma)  is  really  4.5  sigma 
in  the  long  term  as  entropy  sets  in.  Sigma 
shift  translates  to  more  defects  per  million 
-  1,700  times  more.  Statistical  6  sigma  is 
not  3.4  DPMO,  it  is  actually  2  DPBO;  that 
is  defects  per  billion  opportunities,  a  differ¬ 
ence  factor  of  1,700. 

Did  you  just  get  a  sense  of  uneasiness? 
Remember  that  most  companies  claiming 
the  use  of  six  sigma  for  process  improve¬ 
ment  are  not  using  either  of  these  statisti¬ 
cal  values;  they  are  merely  targeting  their 
processes  for  measured  improvement. 

What  if  performance  improved  over 
time,  though,  in  contrast  to  being  subject 
to  entropy?  A  sigma  shift  for  better  would 
be  a  7.5  sigma  process.  A  7.5  sigma 
process  would  have  three  defects  per  hun¬ 
dred  trillion  (3.1  DPhTO  [Schofield  nota¬ 
tion]). 

While  a  7.5  sigma  process  seems  an 
unreasonable  expectation,  at  this  rate  the 
commercial  airline  industry  would 
encounter  a  fatal  event  every  1 7,500  years, 
U.S.  highways  would  incur  23  deaths  per 
year  instead  of  40,000,  and  three  deaths 
per  annum  would  be  realized  from  pre¬ 
scription  defects  instead  of  7,000. 

But  a  7.5  sigma  performance  is  not 
unreasonable  in  the  computing  world. 


Consider  for  a  moment  a  teraflop  machine 
that  operates  at  one  trillion  floating  point 
operations  per  second.  In  a  mere  100  sec¬ 
onds,  three  defects  would  be  generated. 
Within  one  year,  1,246,080  defects  would 
be  generated. 

It  gets  worse.  Within  the  next  year  (or 
so)  the  petaflop  machine  will  be  released. 
A  machine  operating  at  that  speed  could 
generate  over  more  than  one  billion 
defects  per  year  if  operating  at  7.5  sigma. 
Do  you  feel  more  unease?  Do  not  get  pre¬ 
maturely  paranoid  -  a  petaflop  machine  is 
unlikely  to  appear  on  your  desktop  any¬ 
time  soon. 

Fortunately,  hardware  performs  far 
more  reliably  than  the  sigma  levels  just 
described  suggest,  but  that  does  not  apply 
to  software.  Software  defects  cost  the  U.S. 
economy  almost  $60  billion  a  year  [1].  Of 
course,  software  defects  are  not  limited  to 
software.  Auto  companies  such  as  BMW, 
Daimler  Chrysler,  Mitsubishi,  and  Volvo 
have  all  experienced  software-related 
product  malfunctions  (defects)  that 
include  engine  stalls,  wiping  interval  prob¬ 
lems,  gauge  illumination  defects,  and 
transmission  gear  errors  [2].  Software 
technicians  in  Panama  were  charged  with 
murder  after  21  patients  died  from  gamma 
ray  overdoses  in  just  40  months  [3].  Sorry, 
no  sigma  levels  released.  And  yet,  62  per¬ 
cent  of  polled  organizations  lack  a  soft¬ 
ware  quality  assurance  group  [4]. 

Practicing  statistical  something  sigma  is 
an  industry  best  practice.  The  Software 
Engineering  Institute’s  Capability  Maturity 
Model®  Integration  recognizes  the  rele¬ 
vance  of  measurements  and  analysis  by 
placing  it  prominently  as  a  Level  2  Process 
Area  in  its  staged  representation.  Later  in 
the  model,  there  is  the  need  to  identify 
assignable  and  common  cause  variation  at 
maturity  Levels  4  and  5,  respectively. 

So  when  did  statistical  notions  become 
ambiguous  with  words  like  Lean  and  Six 
Sigma?  Perhaps  organizations  should  raise 
an  alert  when  the  term  six  sigma  is  used  to 
investigate  its  contextual  alignment  with 
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When  Did  Six  Sigma  Stop  Being  a  Statistical  Measure? 

expectations,  visions,  and  goals.  Perhaps,  three  types  of  lies  as  lies,  darned  lies  (para- 
too,  the  process  improvement  initiative  phrased),  and  statistics,  but  his  quote 
will  have  an  increased  likelihood  of  sue-  seems  apropos  given  how  Six  Sigma  pro¬ 
cess  -  regardless  of  what  it  is  called.  ponents  use  six  sigma  today.  Six  Sigma 

should  be  reserved  for,  well,  six  sigma  per- 
Conclusion  formance  —  a  statistical  measure  for  varia- 

Given  the  abundance  of  quality  improve-  tion.  Maybe  then  quality  will  translate  to 
ment  and  Six  Sigma  tools  available  to  fewer  product  recalls,  lower  costs  will 
organizations  today,  incorporating  six  mean  that  costs  are  decreased,  and  six 
sigma  measurements  might  not  be  that  sigma  performance  will  equate  to  two 
difficult  —  if  the  organization  chooses  to  defects  per  billion  —  maybe  that  is  asking 
do  so.  For  instance,  brainstorming  tech-  too  much.  Distinguishing  between  statisti- 
niques  for  current  state  weaknesses  could  cally  measured  performance  and  mea- 
be  validated  with  statistical  data  (perhaps  sured  performance  can  help  assess  the 
not  to  a  six  sigma  threshold,  but  the  intro-  true  progress  of  an  improvement  effort, 
duction  of  any  statistical  validation  on  When  applying  Six  Sigma  for  process 
root  cause  analysis  might  provide  relevant  improvement,  do  not  leave  out  the  six 
insight  into  weaknesses).  Root  causes  list-  sigma.^ 
ed  on  cause  and  effect  (Fishbone)  dia¬ 
grams  could  similarly  be  validated  with  References 
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BackTalk 


Win  the  Battle,  Lose  the  War 


Readers  of  my  BackTalk  columns  will  already  know  that  I 
am  a  Civil  War  buff,  and  my  recent  move  to  New  Mexico 
brought  new  sites  to  explore.  I  did  not  realize  that  the  West  was 
the  site  of  many  important  Civil  War  battles.  In  fact,  within  60 
miles  of  Albuquerque,  the  Battle  of  Glorieta  Pass  was  fought. 
This  little-known  battle  has  been  called  the  Gettysburg  of  the  West. 
It  was  such  an  important  Northern  victory  that  it  effectively  sig¬ 
naled  the  end  of  Confederate  influence  in  the  Southwest. 

The  Confederates  were  interested  in  the  West.  They  wanted 
recognition  by  Mexico,  and  they  also  wanted  the  gold  and  silver 
that  was  in  Colorado,  Nevada,  and  California.  Unfortunately  for 
the  Confederates,  New  Mexico  (which  fought  on  the  Union  side) 
stood  in  the  way.  Texas,  a  Confederate  state,  sent  forces  up  the 
Rio  Grande.  The  Confederates  captured  Fort  Fillmore  (near  Las 
Cruces,  N.M.),  proceeded  to  win  the  Battle  of  Valverde,  and 
advanced  north  up  the  Rio  Grande.  The  Confederates  eventual¬ 
ly  occupied  Albuquerque  and  Santa  Fe.  Their  primary  objective 
was  to  take  Fort  Union,  N.M.  —  an  important  Union  federal  sup¬ 
ply  center.  Fort  Union  sat  directly  on  the  Santa  Fe  Trail,  and  is 
about  25  miles  from  present-day  Las  Vegas,  N.M.  It  is  about  50 
miles  northeast  of  Santa  Fe  as  the  crow  flies  (assuming  the  crow 
can  fly  over  pretty  rough  mountain  passes),  but  100  miles  away 
via  the  Santa  Fe  Trail.  To  combat  the  Confederate  advance. 
Union  forces  from  Fort  Union,  augmented  by  a  regiment  of  First 
Colorado  Volunteers,  advanced  south  on  the  Santa  Fe  Trail 
towards  Glorieta  Pass. 

The  Battle  of  Glorieta  Pass  itself  was  fought  from  March  26- 
28,  1862  (the  Battle  of  Gettysburg  also  took  three  days).  For  the 
first  two  days  of  the  battle,  there  was  mixed  fighting  at  several 
locations  near  Santa  Fe  with  inconclusive  results.  On  March  28, 
the  Confederates  advanced  toward  Fort  Union,  initially  heading 
southeast  towards  Glorieta  Pass.  In  what  proved  to  be  a  very 
unfortunate  decision,  the  Confederates  left  all  their  supplies  in  a 
wagon  train  at  Canoncito,  about  halfway  between  Santa  Fe  and 
Glorieta  Pass.  This  wagon  train  was  guarded  by  a  single  cannon 
and  a  handful  of  noncombatants.  This  poorly  defended  supply 
train  (about  eighty  wagons)  contained  the  entire  Confederate 
reserves  of  ammunition,  baggage,  food,  forage,  horses,  mules, 
and  medicines. 

The  Confederate  forces  proceeded  toward  Fort  Union  and 
met  the  advancing  Union  forces  at  Pigeon’s  Rest,  slightly  east  of 
Glorieta  Pass.  The  Confederates  thought  their  supplies  were  safe 
—  after  all,  for  Union  forces  to  reach  the  Confederate  supplies, 
the  Union  forces  would  have  to  go  through  the  Confederates. 
However,  as  the  battle  raged  around  Pigeon’s  Ranch,  a  small 
group  of  Union  forces  were  dispatched  to  find  and  destroy  the 
Confederate  supply  train.  Since  the  Union  forces  consisted  of 
frontiersmen  from  mining  districts  near  Denver,  mountainous 
terrain  did  not  deter  them.  The  Union  dispatchment  avoided  the 
Santa  Fe  Trail,  bypassed  the  Confederate  forces,  and  crossed  over 
16  miles  of  mountainous  terrain.  They  then  located,  attacked, 
and  destroyed  the  entire  Confederate  supply  wagon  train  at 
Canoncito.  The  Union  troops  retraced  their  route  and  rejoined 
the  main  Union  forces  after  dark  (as  the  battle  was  ending). 

The  Confederates  went  to  sleep  that  night  thinking  they  had 
won  this  battle,  just  as  they  had  won  all  their  previous  battles  in 
New  Mexico.  The  Union  forces  also  thought  they  had  won. 

Casualties  on  each  side  were  about  the  same  —  about  50  killed 
and  60  wounded.  But  it  wasn’t  this  relatively  indecisive  battle  that 
was  important.  Unbeknownst  to  the  Confederates,  the  destruc¬ 


tion  of  their  supply  train  checked  the  advance  of  the 
Confederate  forces  in  New  Mexico.  Just  as  the  Battle  of 
Gettysburg  was  the  high-water  mark  of  the  Confederacy ,  the  Battle  of 
Glorieta  Pass  was  definitely  the  high-water  mark  of  the  Confederacy  in 
the  Southwest. 

The  Confederates’  lack  of  supplies  eventually  forced  them  to 
retreat,  backtrack  down  the  Rio  Grande,  and  return  to  San 
Antonio.  It  is  now  recognized  that  the  Battle  of  Glorieta  Pass 
effectively  stopped  a  Confederate  invasion  in  the  Southwest.  The 
battle  signaled  the  end  of  a  valorous  Confederate  presence  along 
the  Rio  Grande  in  the  War  of  Northern  Aggression  (I  AM  a 
Southerner  and  a  Texas  A&M  grad  —  it  was  either  put  this  in  or 
get  tarred  and  feathered  at  my  next  family  or  class  reunion). 

What  does  all  this  possibly  have  to  do  with  maturity  models? 
Because  simply  developing  code  is  like  fighting  a  battle.  Winning 
one  battle  is  not  enough.  To  win  the  war,  you  need  to  be  able  to 
fully  support  all  your  assets.  In  the  battle  for  the  Southwest  along 
the  Rio  Grande,  the  Confederates  won  most,  if  not  all,  of  the 
battles.  However,  it  was  the  lack  of  support  and  assets  that  cost 
them  the  war. 

It  doesn’t  really  do  you  any  good  to  deliver  code  to  your  users 
if  you  cannot  provide  support  for  maintenance  and  updates.  You 
need  a  process  (or  maturity  model)  in  place  to  ensure  that  you 
can  provide  long-time  support.  Just  as  losing  the  supply  train 
spelled  the  end  of  the  Confederacy  along  the  Rio  Grande,  prob¬ 
lems  with  life-cycle  support  can  easily  spell  the  end  of  your 
development  effort. 

Maturity  models  have  to  support  everything  you  are  going  to 
need  to  eventually  win  the  war,  not  just  the  upcoming  batde.  If  it 
doesn’t,  then  you  need  to  implement  whatever  process  it  takes.  If 
you  are  only  worried  about  the  current  skirmish,  ask  yourself  this: 
“Is  my  lack  of  configuration  management,  risk  management,  or 
requirements  management  going  to  eventually  cost  me  the  victory?” 
Short-term  thinking  wins  battles.  Long-term  thinking  wins  wars. 

Do  you  have  a  maturity  model  in  place?  Does  it  work?  If 
not,  then  why  aren’t  you  fixing  it? 

Do  you  want  to  win  a  battle,  or  win  the  war? 

—  David  A.  Cook,  Ph.D. 

Senior  Research  Scientist  (and  Civil  War  Buff) 
The  AEgis  Technologies  Group,  Inc. 

dcook@aegistg.com 

Additional  Reading 

1.  See  <www.santafetrailnm.org>,  <http://web.archive.org/ 
web/20001 002020035>,  <www.nmhu.edu/researchsftrail/ 
mapsftl  .gif>,  <http: / / web.archive.org/ web / 2000 1 002020035 / >, 
and  <www.nmhu.edu/ research/ sftrail/ mapsftl  .gif>  for 
maps  and  more  information  on  Glorieta  Pass. 

2.  Information  in  the  preceding  paragraphs  has  been  taken  from 
an  article  by  Don  E.  Alberts  at  <www.tsha.utexas.edu/ 
handbook/ online/ articles/ GG/ qfg2.html>,  <http: / / www. 
tsha.utexas.edu/handbook/ online/ articles/ GG/ qfg2.html>, 
and  also  from  <http://history.sandiego.edu/gen/civilwar/ 
14/glorieta.html>  and  <http://history.sandiego.edugen/ 
civilwar/14/glorieta.html>.  Additional  information  has  been 
taken  from  personal  trips  to  New  Mexico’s  Las  Cruces, 
Glorieta  Pass,  Pecos  National  Historical  Park,  and  Ft  Union 
National  Monument.  And  no,  I  am  not  in  the  employ  of  the 
New  Mexico  Department  of  Tourism. 
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